[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:30:05 dist-test-slave-qp2n krb5kdc[15906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:30:05 dist-test-slave-qp2n krb5kdc[15906](info): set up 2 sockets
Sep 28 08:30:05 dist-test-slave-qp2n krb5kdc[15906](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 08:30:06.484154 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.005s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:30:06 dist-test-slave-qp2n krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:30:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:30:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:30:06.677320 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:34085
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:34085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:07.179199 15919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:07.179508 15919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:07.179665 15919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:07.189079 15919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:30:07.189250 15919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:07.189419 15919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:07.189642 15919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:30:07.189805 15919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:30:07.202396 15919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:34085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:34085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:07.203864 15919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:07.205332 15919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:07.226445 15927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:07.226496 15925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:07.226805 15924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:07.227990 15919 server_base.cc:1034] running on GCE node
I20260928 08:30:07.229003 15919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:07.230720 15919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:07.231987 15919 hybrid_clock.cc:648] HybridClock initialized: now 1790584207231942 us; error 94 us; skew 500 ppm
I20260928 08:30:07.232424 15919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:07 dist-test-slave-qp2n krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584207, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:07.240477 15919 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:30:07.242314 15919 webserver.cc:460] Webserver started at http://127.15.134.254:46815/ using document root <none> and password file <none>
I20260928 08:30:07.243021 15919 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:07.243137 15919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:07.243398 15919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:07.246490 15919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "2f0405852c7f40678d6b6f0e714a779d"
format_stamp: "Formatted at 2026-09-28 08:30:07 on dist-test-slave-qp2n"
server_key: "34ec81f26574365e86f2eecd40d63cdf"
server_key_iv: "0ad2bf39beda5ae2654dd1140086cfe1"
server_key_version: "encryptionkey@0"
I20260928 08:30:07.247299 15919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "2f0405852c7f40678d6b6f0e714a779d"
format_stamp: "Formatted at 2026-09-28 08:30:07 on dist-test-slave-qp2n"
server_key: "34ec81f26574365e86f2eecd40d63cdf"
server_key_iv: "0ad2bf39beda5ae2654dd1140086cfe1"
server_key_version: "encryptionkey@0"
I20260928 08:30:07.254227 15919 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260928 08:30:07.259016 15934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:07.260497 15919 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260928 08:30:07.260715 15919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "2f0405852c7f40678d6b6f0e714a779d"
format_stamp: "Formatted at 2026-09-28 08:30:07 on dist-test-slave-qp2n"
server_key: "34ec81f26574365e86f2eecd40d63cdf"
server_key_iv: "0ad2bf39beda5ae2654dd1140086cfe1"
server_key_version: "encryptionkey@0"
I20260928 08:30:07.261037 15919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:07.277659 15919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:07.283327 15919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:07.283643 15919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:07.312824 15919 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:34085
I20260928 08:30:07.312885 15985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:34085 every 8 connection(s)
I20260928 08:30:07.314560 15919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:30:07.319228 15986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:07.319180 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 15919
I20260928 08:30:07.319598 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:30:07.322336 15899 external_mini_cluster.cc:1428] Setting key 1ec6abd84f5e1c74acd8c4e76afc16f5
I20260928 08:30:07.332964 15986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f0405852c7f40678d6b6f0e714a779d: Bootstrap starting.
I20260928 08:30:07.335875 15986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f0405852c7f40678d6b6f0e714a779d: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:07.337085 15986 log.cc:824] T 00000000000000000000000000000000 P 2f0405852c7f40678d6b6f0e714a779d: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:07.341454 15986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f0405852c7f40678d6b6f0e714a779d: No bootstrap required, opened a new log
I20260928 08:30:07.351481 15986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f0405852c7f40678d6b6f0e714a779d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0405852c7f40678d6b6f0e714a779d" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 34085 } }
I20260928 08:30:07.351924 15986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f0405852c7f40678d6b6f0e714a779d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:07.352052 15986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f0405852c7f40678d6b6f0e714a779d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f0405852c7f40678d6b6f0e714a779d, State: Initialized, Role: FOLLOWER
I20260928 08:30:07.352752 15986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f0405852c7f40678d6b6f0e714a779d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0405852c7f40678d6b6f0e714a779d" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 34085 } }
I20260928 08:30:07.353044 15986 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f0405852c7f40678d6b6f0e714a779d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:30:07.353204 15986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f0405852c7f40678d6b6f0e714a779d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:30:07.353400 15986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f0405852c7f40678d6b6f0e714a779d [term 0 FOLLOWER]: Advancing to term 1
Sep 28 08:30:07 dist-test-slave-qp2n krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:07.356765 15986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f0405852c7f40678d6b6f0e714a779d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0405852c7f40678d6b6f0e714a779d" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 34085 } }
I20260928 08:30:07.357519 15986 leader_election.cc:302] T 00000000000000000000000000000000 P 2f0405852c7f40678d6b6f0e714a779d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f0405852c7f40678d6b6f0e714a779d; no voters:
I20260928 08:30:07.361021 15986 leader_election.cc:288] T 00000000000000000000000000000000 P 2f0405852c7f40678d6b6f0e714a779d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:30:07.361249 15991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f0405852c7f40678d6b6f0e714a779d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:07.363243 15991 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f0405852c7f40678d6b6f0e714a779d [term 1 LEADER]: Becoming Leader. State: Replica: 2f0405852c7f40678d6b6f0e714a779d, State: Running, Role: LEADER
I20260928 08:30:07.363945 15991 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f0405852c7f40678d6b6f0e714a779d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0405852c7f40678d6b6f0e714a779d" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 34085 } }
I20260928 08:30:07.364114 15986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f0405852c7f40678d6b6f0e714a779d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:30:07.368116 15992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f0405852c7f40678d6b6f0e714a779d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f0405852c7f40678d6b6f0e714a779d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0405852c7f40678d6b6f0e714a779d" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 34085 } } }
I20260928 08:30:07.368680 15992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f0405852c7f40678d6b6f0e714a779d [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:07.372730 15988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:07.327300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41908 (local address 127.15.134.254:34085)
0928 08:30:07.328390 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:07.328405 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:07.330023 (+ 1618us) server_negotiation.cc:407] Connection header received
0928 08:30:07.340575 (+ 10552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:07.340603 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:07.341118 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:07.341576 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:07.346043 (+ 4467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:07.347527 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:07.349456 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:07.350274 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:07.350479 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:07.357006 (+ 6527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:07.357020 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:07.357040 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:07.357111 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:07.362899 (+ 5788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:07.364286 (+ 1387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:07.364295 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:07.364302 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:07.364572 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:07.365297 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:07.365302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:07.365462 (+ 160us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:07.366394 (+ 932us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:07.366843 (+ 449us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:07.370372 (+ 3529us) server_negotiation.cc:299] Negotiation successful
0928 08:30:07.370920 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":810,"thread_start_us":510,"threads_started":1}
I20260928 08:30:07.372718 15993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f0405852c7f40678d6b6f0e714a779d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f0405852c7f40678d6b6f0e714a779d. Latest consensus state: current_term: 1 leader_uuid: "2f0405852c7f40678d6b6f0e714a779d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0405852c7f40678d6b6f0e714a779d" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 34085 } } }
I20260928 08:30:07.373111 15993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f0405852c7f40678d6b6f0e714a779d [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:07.375948 15996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:30:07.384351 15996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:30:07.395159 15996 catalog_manager.cc:1347] Generated new cluster ID: b96f571d9c324350ac8f3752c8d95629
I20260928 08:30:07.395334 15996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:30:07.409855 15996 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:30:07.412016 15996 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:30:07.424736 15996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f0405852c7f40678d6b6f0e714a779d: Generated new TSK 0
I20260928 08:30:07.425720 15996 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.134.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
I20260928 08:30:07.469503 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:0
--local_ip_for_outbound_sockets=127.15.134.193
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:34085
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:07.928881 16012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:07.929231 16012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:07.929409 16012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:07.937717 16012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:07.937901 16012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:07.938081 16012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:30:07.949489 16012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:34085
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:07.951098 16012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:07.952589 16012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:07.967825 16020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:07.968330 16017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:07.968401 16018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:07.969293 16012 server_base.cc:1034] running on GCE node
I20260928 08:30:07.970086 16012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:07.971582 16012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:07.972800 16012 hybrid_clock.cc:648] HybridClock initialized: now 1790584207972726 us; error 69 us; skew 500 ppm
I20260928 08:30:07.973120 16012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:07 dist-test-slave-qp2n krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584207, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:07.976923 16012 init.cc:377] Logged in from keytab as kudu/127.15.134.193@KRBTEST.COM (short username kudu)
I20260928 08:30:07.978719 16012 webserver.cc:460] Webserver started at http://127.15.134.193:43093/ using document root <none> and password file <none>
I20260928 08:30:07.979478 16012 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:07.979609 16012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:07.979970 16012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:07.983273 16012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-0/data/instance:
uuid: "b722679e82464502907f45ee5451ddef"
format_stamp: "Formatted at 2026-09-28 08:30:07 on dist-test-slave-qp2n"
server_key: "3ddbd13ff03158544bd21d940c22ddea"
server_key_iv: "f074f704d732c93e2d7debd47e8c7d82"
server_key_version: "encryptionkey@0"
I20260928 08:30:07.984102 16012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance:
uuid: "b722679e82464502907f45ee5451ddef"
format_stamp: "Formatted at 2026-09-28 08:30:07 on dist-test-slave-qp2n"
server_key: "3ddbd13ff03158544bd21d940c22ddea"
server_key_iv: "f074f704d732c93e2d7debd47e8c7d82"
server_key_version: "encryptionkey@0"
I20260928 08:30:07.990127 16012 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260928 08:30:07.995067 16027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:07.996724 16012 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 08:30:07.996914 16012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "b722679e82464502907f45ee5451ddef"
format_stamp: "Formatted at 2026-09-28 08:30:07 on dist-test-slave-qp2n"
server_key: "3ddbd13ff03158544bd21d940c22ddea"
server_key_iv: "f074f704d732c93e2d7debd47e8c7d82"
server_key_version: "encryptionkey@0"
I20260928 08:30:07.997150 16012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:08.017078 16012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:08.022411 16012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:08.022730 16012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:08.023931 16012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:30:08.025677 16012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:08.025779 16012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:08.025945 16012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:08.026130 16012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:08.075307 16012 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:39999
I20260928 08:30:08.075598 16139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:39999 every 8 connection(s)
I20260928 08:30:08.077116 16012 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:30:08.082193 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 16012
I20260928 08:30:08.082491 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance
I20260928 08:30:08.083611 15899 external_mini_cluster.cc:1428] Setting key 17f1fb15da1b727e61f837be2608f7c0
Sep 28 08:30:08 dist-test-slave-qp2n krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584207, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.134.194@KRBTEST.COM; defaulting to no policy
I20260928 08:30:08.102325 16142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:08.080645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:44237 (local address 127.15.134.254:34085)
0928 08:30:08.081174 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:08.081179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:08.082893 (+ 1714us) server_negotiation.cc:407] Connection header received
0928 08:30:08.085750 (+ 2857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:08.085756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:08.085844 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:08.085963 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:08.087927 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:08.088780 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:08.090331 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:08.091012 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:08.091190 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:08.095822 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:08.095827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:08.095831 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:08.095864 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:08.098886 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:08.099629 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:08.099637 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:08.099642 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:08.099768 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:08.100202 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:08.100207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:08.100211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:08.100590 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:08.100794 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:08.101728 (+ 934us) server_negotiation.cc:299] Negotiation successful
0928 08:30:08.101934 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":281,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.194@KRBTEST.COM" created.
I20260928 08:30:08.103735 16143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:08.081321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:34085 (local address 127.15.134.193:44237)
0928 08:30:08.082638 (+ 1317us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:08.082703 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:08.085472 (+ 2769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:08.086282 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:08.086291 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:08.086850 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:08.087679 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:08.087701 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:08.088937 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:08.088943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:08.090182 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:08.090193 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:08.091277 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:08.091285 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:08.091539 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:08.092418 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:08.092489 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:08.095596 (+ 3107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:08.099031 (+ 3435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:08.099040 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:08.099064 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:08.099507 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:08.099882 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:08.099888 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:08.099891 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:08.100077 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:08.100768 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:08.100775 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:08.101242 (+ 467us) client_negotiation.cc:769] Sending connection context
0928 08:30:08.101680 (+ 438us) client_negotiation.cc:240] Negotiation successful
0928 08:30:08.102165 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"thread_start_us":718,"threads_started":1}
I20260928 08:30:08.105389 16140 heartbeater.cc:347] Connected to a master server at 127.15.134.254:34085
I20260928 08:30:08.105986 16140 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:08.106938 16140 heartbeater.cc:511] Master 127.15.134.254:34085 requested a full tablet report, sending...
I20260928 08:30:08.109436 15951 ts_manager.cc:194] Registered new tserver with Master: b722679e82464502907f45ee5451ddef (127.15.134.193:39999)
I20260928 08:30:08.110891 15951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.193@KRBTEST.COM'} at 127.15.134.193:44237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
I20260928 08:30:08.127789 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:0
--local_ip_for_outbound_sockets=127.15.134.194
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:34085
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:08.320564 15982 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.15.134.254:34085
W20260928 08:30:08.550153 16146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:08.550514 16146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:08.550725 16146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:08.558602 16146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:08.558727 16146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:08.559021 16146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:30:08.571463 16146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:34085
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:08.573130 16146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:08.574856 16146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:08.591979 16151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:08.592159 16154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:08.592315 16152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:08.594149 16146 server_base.cc:1034] running on GCE node
I20260928 08:30:08.595120 16146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:08.596233 16146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:08.597528 16146 hybrid_clock.cc:648] HybridClock initialized: now 1790584208597407 us; error 51 us; skew 500 ppm
I20260928 08:30:08.597954 16146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:08 dist-test-slave-qp2n krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584208, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:08.602205 16146 init.cc:377] Logged in from keytab as kudu/127.15.134.194@KRBTEST.COM (short username kudu)
I20260928 08:30:08.604735 16146 webserver.cc:460] Webserver started at http://127.15.134.194:34639/ using document root <none> and password file <none>
I20260928 08:30:08.605530 16146 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:08.605702 16146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:08.606056 16146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:08.609560 16146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-1/data/instance:
uuid: "d8d135786b0b4088bed35d8dc1947e50"
format_stamp: "Formatted at 2026-09-28 08:30:08 on dist-test-slave-qp2n"
server_key: "e592b4f0feaf28ea2378b5357c486d7b"
server_key_iv: "db4ed737a62a8807148cda0a289395b0"
server_key_version: "encryptionkey@0"
I20260928 08:30:08.610363 16146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance:
uuid: "d8d135786b0b4088bed35d8dc1947e50"
format_stamp: "Formatted at 2026-09-28 08:30:08 on dist-test-slave-qp2n"
server_key: "e592b4f0feaf28ea2378b5357c486d7b"
server_key_iv: "db4ed737a62a8807148cda0a289395b0"
server_key_version: "encryptionkey@0"
I20260928 08:30:08.616989 16146 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 08:30:08.621316 16161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:08.622852 16146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:30:08.623139 16146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "d8d135786b0b4088bed35d8dc1947e50"
format_stamp: "Formatted at 2026-09-28 08:30:08 on dist-test-slave-qp2n"
server_key: "e592b4f0feaf28ea2378b5357c486d7b"
server_key_iv: "db4ed737a62a8807148cda0a289395b0"
server_key_version: "encryptionkey@0"
I20260928 08:30:08.623406 16146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:08.637450 16146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:08.642299 16146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:08.642616 16146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:08.643646 16146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:30:08.645258 16146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:08.645362 16146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:08.645596 16146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:08.645694 16146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:08.690605 16146 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:39661
I20260928 08:30:08.690761 16273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:39661 every 8 connection(s)
I20260928 08:30:08.692062 16146 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:30:08.698007 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 16146
I20260928 08:30:08.698261 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance
I20260928 08:30:08.698741 15899 external_mini_cluster.cc:1428] Setting key cfb89edad48502c009529f1f56624751
Sep 28 08:30:08 dist-test-slave-qp2n krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584208, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.134.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.195@KRBTEST.COM" created.
I20260928 08:30:08.722487 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:08.695722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:33073 (local address 127.15.134.254:34085)
0928 08:30:08.696276 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:08.696282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:08.697505 (+ 1223us) server_negotiation.cc:407] Connection header received
0928 08:30:08.701027 (+ 3522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:08.701033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:08.701116 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:08.701239 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:08.703677 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:08.705378 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:08.708517 (+ 3139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:08.709149 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:08.709253 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:08.715076 (+ 5823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:08.715081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:08.715085 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:08.715127 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:08.718368 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:08.719254 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:08.719262 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:08.719270 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:08.719435 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:08.720054 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:08.720058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:08.720060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:08.720400 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:08.720512 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:08.721621 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0928 08:30:08.721912 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":311,"threads_started":1}
I20260928 08:30:08.723238 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:08.696373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:34085 (local address 127.15.134.194:33073)
0928 08:30:08.697301 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:08.697342 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:08.700701 (+ 3359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:08.701517 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:08.701532 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:08.702501 (+ 969us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:08.703495 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:08.703516 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:08.705586 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:08.705596 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:08.708329 (+ 2733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:08.708357 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:08.709454 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:08.709465 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:08.709886 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:08.710822 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:08.710859 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:08.714775 (+ 3916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:08.718552 (+ 3777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:08.718563 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:08.718585 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:08.719111 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:08.719595 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:08.719600 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:08.719602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:08.719939 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:08.720552 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:08.720559 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:08.721131 (+ 572us) client_negotiation.cc:769] Sending connection context
0928 08:30:08.721482 (+ 351us) client_negotiation.cc:240] Negotiation successful
0928 08:30:08.721924 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":447,"threads_started":1}
I20260928 08:30:08.725073 16274 heartbeater.cc:347] Connected to a master server at 127.15.134.254:34085
I20260928 08:30:08.725492 16274 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:08.726429 16274 heartbeater.cc:511] Master 127.15.134.254:34085 requested a full tablet report, sending...
I20260928 08:30:08.728515 15951 ts_manager.cc:194] Registered new tserver with Master: d8d135786b0b4088bed35d8dc1947e50 (127.15.134.194:39661)
I20260928 08:30:08.729506 15951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.194@KRBTEST.COM'} at 127.15.134.194:33073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
I20260928 08:30:08.746654 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:0
--local_ip_for_outbound_sockets=127.15.134.195
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:34085
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:34037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:30:09.115118 16140 heartbeater.cc:503] Master 127.15.134.254:34085 was elected leader, sending a full tablet report...
W20260928 08:30:09.149626 16280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:09.149940 16280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:09.150038 16280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:09.157405 16280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:09.157521 16280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:09.157635 16280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:30:09.168160 16280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:34037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:34085
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:09.169471 16280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:09.170790 16280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:09.184726 16285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:09.184733 16286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:09.185043 16288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:09.185657 16280 server_base.cc:1034] running on GCE node
I20260928 08:30:09.186329 16280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:09.187291 16280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:09.188504 16280 hybrid_clock.cc:648] HybridClock initialized: now 1790584209188439 us; error 63 us; skew 500 ppm
I20260928 08:30:09.188898 16280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:09 dist-test-slave-qp2n krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584209, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:09.192077 16280 init.cc:377] Logged in from keytab as kudu/127.15.134.195@KRBTEST.COM (short username kudu)
I20260928 08:30:09.193694 16280 webserver.cc:460] Webserver started at http://127.15.134.195:36433/ using document root <none> and password file <none>
I20260928 08:30:09.194502 16280 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:09.194653 16280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:09.195005 16280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:09.198081 16280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-2/data/instance:
uuid: "00b092e63f254108934d44f6d3a591a7"
format_stamp: "Formatted at 2026-09-28 08:30:09 on dist-test-slave-qp2n"
server_key: "7f3c7a4d9c7f55c85d3e4e15d2d32b73"
server_key_iv: "cd13bef6b8c6439716904c58e93aafed"
server_key_version: "encryptionkey@0"
I20260928 08:30:09.198967 16280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance:
uuid: "00b092e63f254108934d44f6d3a591a7"
format_stamp: "Formatted at 2026-09-28 08:30:09 on dist-test-slave-qp2n"
server_key: "7f3c7a4d9c7f55c85d3e4e15d2d32b73"
server_key_iv: "cd13bef6b8c6439716904c58e93aafed"
server_key_version: "encryptionkey@0"
I20260928 08:30:09.204902 16280 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 08:30:09.208853 16295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:09.210309 16280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:30:09.210558 16280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "00b092e63f254108934d44f6d3a591a7"
format_stamp: "Formatted at 2026-09-28 08:30:09 on dist-test-slave-qp2n"
server_key: "7f3c7a4d9c7f55c85d3e4e15d2d32b73"
server_key_iv: "cd13bef6b8c6439716904c58e93aafed"
server_key_version: "encryptionkey@0"
I20260928 08:30:09.210794 16280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:09.225212 16280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:09.229797 16280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:09.230157 16280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:09.231096 16280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:30:09.232506 16280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:09.232658 16280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:09.232812 16280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:09.232887 16280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:09.279757 16280 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:43311
I20260928 08:30:09.279801 16407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:43311 every 8 connection(s)
I20260928 08:30:09.281119 16280 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:30:09.286223 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 16280
I20260928 08:30:09.286522 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance
I20260928 08:30:09.287101 15899 external_mini_cluster.cc:1428] Setting key 55165067b6557fe27714643ff8f90159
Sep 28 08:30:09 dist-test-slave-qp2n krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584209, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:09.303551 16410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:09.284285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:38671 (local address 127.15.134.254:34085)
0928 08:30:09.285029 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:09.285034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:09.285957 (+ 923us) server_negotiation.cc:407] Connection header received
0928 08:30:09.287899 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:09.287917 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:09.288033 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:09.288162 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:09.290140 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:09.291198 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:09.292934 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:09.293387 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:09.293556 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:09.297141 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:09.297146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:09.297149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:09.297181 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:09.299891 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:09.300755 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:09.300760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:09.300763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:09.300961 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:09.301433 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:09.301440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:09.301444 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:09.301775 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:09.301961 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:09.302894 (+ 933us) server_negotiation.cc:299] Negotiation successful
0928 08:30:09.303145 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":413,"threads_started":1}
I20260928 08:30:09.304679 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:09.284696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:34085 (local address 127.15.134.195:38671)
0928 08:30:09.285751 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:09.285804 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:09.287632 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:09.288424 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:09.288432 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:09.289284 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:09.289945 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:09.289966 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:09.291349 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:09.291355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:09.292777 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:09.292798 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:09.293675 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:09.293682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:09.293921 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:09.294622 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:09.294644 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:09.296912 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:09.300072 (+ 3160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:09.300082 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:09.300104 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:09.300565 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:09.301089 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:09.301095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:09.301099 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:09.301296 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:09.301935 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:09.301944 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:09.302505 (+ 561us) client_negotiation.cc:769] Sending connection context
0928 08:30:09.302794 (+ 289us) client_negotiation.cc:240] Negotiation successful
0928 08:30:09.303352 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":420,"threads_started":1}
I20260928 08:30:09.306316 16408 heartbeater.cc:347] Connected to a master server at 127.15.134.254:34085
I20260928 08:30:09.306726 16408 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:09.307526 16408 heartbeater.cc:511] Master 127.15.134.254:34085 requested a full tablet report, sending...
I20260928 08:30:09.309201 15951 ts_manager.cc:194] Registered new tserver with Master: 00b092e63f254108934d44f6d3a591a7 (127.15.134.195:43311)
I20260928 08:30:09.310097 15951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.195@KRBTEST.COM'} at 127.15.134.195:38671
I20260928 08:30:09.316160 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 08:30:09 dist-test-slave-qp2n krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584209, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 08:30:09 dist-test-slave-qp2n krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584209, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.134.193@KRBTEST.COM
I20260928 08:30:09.369036 16416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:09.347731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42224 (local address 127.15.134.193:39999)
0928 08:30:09.348792 (+ 1061us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:09.348816 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:09.349125 (+ 309us) server_negotiation.cc:407] Connection header received
0928 08:30:09.349385 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:09.349403 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:09.349646 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:09.349879 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:09.352976 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:09.354355 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:09.355900 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:09.356803 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:09.356940 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:09.361917 (+ 4977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:09.361932 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:09.361948 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:09.361997 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:09.365804 (+ 3807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:09.366665 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:09.366674 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:09.366676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:09.366904 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:09.367325 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:09.367329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:09.367331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:09.367752 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:09.368106 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:09.368263 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 08:30:09.368557 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":783,"thread_start_us":491,"threads_started":1}
W20260928 08:30:09.370442 16055 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42224
Sep 28 08:30:09 dist-test-slave-qp2n krb5kdc[15906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 08:30:09 dist-test-slave-qp2n krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.193@KRBTEST.COM
I20260928 08:30:09.417146 16416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:09.401743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42228 (local address 127.15.134.193:39999)
0928 08:30:09.402007 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:09.402016 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:09.402397 (+ 381us) server_negotiation.cc:407] Connection header received
0928 08:30:09.402604 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:09.402608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:09.402674 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:09.402784 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:09.404177 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:09.404950 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:09.406172 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:09.406589 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:09.406721 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:09.411088 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:09.411093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:09.411096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:09.411128 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:09.414557 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:09.415204 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:09.415212 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:09.415219 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:09.415347 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:09.415734 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:09.415738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:09.415740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:09.416026 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:09.416200 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:09.416604 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 08:30:09.416830 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 08:30:09.418820 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 16012
I20260928 08:30:09.433516 16416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:09.419745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42232 (local address 127.15.134.193:39999)
0928 08:30:09.419893 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:09.419897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:09.420523 (+ 626us) server_negotiation.cc:407] Connection header received
0928 08:30:09.420692 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:09.420695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:09.420747 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:09.420821 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:09.422319 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:09.423102 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:09.424728 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:09.425336 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:09.425434 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:09.427990 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:09.427998 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:09.428002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:09.428043 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:09.430767 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:09.431507 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:09.431512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:09.431514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:09.431798 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:09.432169 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:09.432172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:09.432174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:09.432531 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:09.432797 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:09.433030 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 08:30:09.433180 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 08:30:09.434249 16135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:09.520215 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 16012
I20260928 08:30:09.538465 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 16146
Sep 28 08:30:09 dist-test-slave-qp2n krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.194@KRBTEST.COM
I20260928 08:30:09.555300 16423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:09.539728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48316 (local address 127.15.134.194:39661)
0928 08:30:09.540392 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:09.540400 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:09.540427 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:30:09.540488 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:09.540495 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:09.540788 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:09.540922 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:09.542221 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:09.543205 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:09.544460 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:09.545089 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:09.545195 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:09.549571 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:09.549585 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:09.549601 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:09.549636 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:09.552472 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:09.553261 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:09.553269 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:09.553271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:09.553394 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:09.553816 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:09.553819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:09.553821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:09.554170 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:09.554401 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:09.554778 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 08:30:09.554978 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":374,"threads_started":1}
I20260928 08:30:09.556906 16269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:09.641636 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 16146
I20260928 08:30:09.665225 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 16280
Sep 28 08:30:09 dist-test-slave-qp2n krb5kdc[15906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.195@KRBTEST.COM
I20260928 08:30:09.682451 16425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:09.666454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54430 (local address 127.15.134.195:43311)
0928 08:30:09.667018 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:09.667026 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:09.667060 (+ 34us) server_negotiation.cc:407] Connection header received
0928 08:30:09.667119 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:09.667125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:09.667337 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:09.667515 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:09.668936 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:09.669972 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:09.671260 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:09.671832 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:09.671984 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:09.676364 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:09.676383 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:09.676399 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:09.676435 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:09.679455 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:09.680234 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:09.680241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:09.680244 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:09.680363 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:09.680801 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:09.680807 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:09.680810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:09.681178 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:09.681507 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:09.681810 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 08:30:09.682099 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":279,"threads_started":1}
I20260928 08:30:09.683816 16403 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:09.767483 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 16280
I20260928 08:30:09.785058 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 15919
I20260928 08:30:09.786507 15981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:09.890281 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 15919
2026-09-28T08:30:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4572 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:30:09 dist-test-slave-qp2n krb5kdc[16430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:30:09 dist-test-slave-qp2n krb5kdc[16430](info): set up 2 sockets
Sep 28 08:30:09 dist-test-slave-qp2n krb5kdc[16430](info): commencing operation
krb5kdc: starting...
W20260928 08:30:11.007283 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:30:11 dist-test-slave-qp2n krb5kdc[16430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:30:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:30:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:30:11.198478 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:35479
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:41759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:35479
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:11.545820 16443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:11.546165 16443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:11.546260 16443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:11.553828 16443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:30:11.553951 16443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:11.553997 16443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:11.554051 16443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:30:11.554095 16443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:30:11.565073 16443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:41759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:35479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:35479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:11.566568 16443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:11.567971 16443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:11.580901 16448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:11.580865 16449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:11.581090 16451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:11.581753 16443 server_base.cc:1034] running on GCE node
I20260928 08:30:11.582599 16443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:11.584146 16443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:11.585389 16443 hybrid_clock.cc:648] HybridClock initialized: now 1790584211585327 us; error 55 us; skew 500 ppm
I20260928 08:30:11.585814 16443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:11 dist-test-slave-qp2n krb5kdc[16430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584211, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:11.589099 16443 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:30:11.590799 16443 webserver.cc:460] Webserver started at http://127.15.134.254:45219/ using document root <none> and password file <none>
I20260928 08:30:11.591535 16443 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:11.591701 16443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:11.592031 16443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:11.595185 16443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "006abbefcdac41d1915f9cc379f9c3a7"
format_stamp: "Formatted at 2026-09-28 08:30:11 on dist-test-slave-qp2n"
server_key: "1c8d97f6b35eac16f3caf77f67509da7"
server_key_iv: "5c58eb1b244ae607ab2d981c2a2db68f"
server_key_version: "encryptionkey@0"
I20260928 08:30:11.595999 16443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "006abbefcdac41d1915f9cc379f9c3a7"
format_stamp: "Formatted at 2026-09-28 08:30:11 on dist-test-slave-qp2n"
server_key: "1c8d97f6b35eac16f3caf77f67509da7"
server_key_iv: "5c58eb1b244ae607ab2d981c2a2db68f"
server_key_version: "encryptionkey@0"
I20260928 08:30:11.601543 16443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 08:30:11.605497 16458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:11.606950 16443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 08:30:11.607142 16443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "006abbefcdac41d1915f9cc379f9c3a7"
format_stamp: "Formatted at 2026-09-28 08:30:11 on dist-test-slave-qp2n"
server_key: "1c8d97f6b35eac16f3caf77f67509da7"
server_key_iv: "5c58eb1b244ae607ab2d981c2a2db68f"
server_key_version: "encryptionkey@0"
I20260928 08:30:11.607352 16443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:11.637135 16443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:11.641947 16443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:11.642287 16443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:11.662057 16443 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:35479
I20260928 08:30:11.662079 16509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:35479 every 8 connection(s)
I20260928 08:30:11.663552 16443 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:30:11.667132 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 16443
I20260928 08:30:11.667384 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:30:11.668161 16510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:11.668249 15899 external_mini_cluster.cc:1428] Setting key 36a7bddc9974863cd9e0dd554d7ab78d
Sep 28 08:30:11 dist-test-slave-qp2n krb5kdc[16430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:11.683720 16510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 006abbefcdac41d1915f9cc379f9c3a7: Bootstrap starting.
I20260928 08:30:11.687247 16510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 006abbefcdac41d1915f9cc379f9c3a7: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:11.688383 16510 log.cc:824] T 00000000000000000000000000000000 P 006abbefcdac41d1915f9cc379f9c3a7: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:11.691553 16510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 006abbefcdac41d1915f9cc379f9c3a7: No bootstrap required, opened a new log
I20260928 08:30:11.692731 16513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:11.670465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.15.134.254:35479)
0928 08:30:11.671290 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:11.671302 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:11.671345 (+ 43us) server_negotiation.cc:407] Connection header received
0928 08:30:11.673238 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:11.673261 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:11.673673 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:11.674027 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:11.675939 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:11.677349 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:11.679130 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:11.679858 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:11.680046 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:11.684955 (+ 4909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:11.684965 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:11.684982 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:11.685026 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:11.688270 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:11.689105 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:11.689114 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:11.689120 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:11.689309 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:11.689706 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:11.689712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:11.689715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:11.690262 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:11.690646 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:11.691154 (+ 508us) server_negotiation.cc:299] Negotiation successful
0928 08:30:11.691592 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":352,"threads_started":1}
I20260928 08:30:11.700425 16510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 006abbefcdac41d1915f9cc379f9c3a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006abbefcdac41d1915f9cc379f9c3a7" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35479 } }
I20260928 08:30:11.700943 16510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 006abbefcdac41d1915f9cc379f9c3a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:11.701027 16510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 006abbefcdac41d1915f9cc379f9c3a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 006abbefcdac41d1915f9cc379f9c3a7, State: Initialized, Role: FOLLOWER
I20260928 08:30:11.701639 16510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 006abbefcdac41d1915f9cc379f9c3a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006abbefcdac41d1915f9cc379f9c3a7" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35479 } }
I20260928 08:30:11.701850 16510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 006abbefcdac41d1915f9cc379f9c3a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:30:11.701943 16510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 006abbefcdac41d1915f9cc379f9c3a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:30:11.702054 16510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 006abbefcdac41d1915f9cc379f9c3a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:11.704596 16510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 006abbefcdac41d1915f9cc379f9c3a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006abbefcdac41d1915f9cc379f9c3a7" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35479 } }
I20260928 08:30:11.705114 16510 leader_election.cc:302] T 00000000000000000000000000000000 P 006abbefcdac41d1915f9cc379f9c3a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 006abbefcdac41d1915f9cc379f9c3a7; no voters:
I20260928 08:30:11.705693 16510 leader_election.cc:288] T 00000000000000000000000000000000 P 006abbefcdac41d1915f9cc379f9c3a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:30:11.705842 16515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 006abbefcdac41d1915f9cc379f9c3a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:11.706566 16515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 006abbefcdac41d1915f9cc379f9c3a7 [term 1 LEADER]: Becoming Leader. State: Replica: 006abbefcdac41d1915f9cc379f9c3a7, State: Running, Role: LEADER
I20260928 08:30:11.707137 16515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 006abbefcdac41d1915f9cc379f9c3a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006abbefcdac41d1915f9cc379f9c3a7" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35479 } }
I20260928 08:30:11.708127 16510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 006abbefcdac41d1915f9cc379f9c3a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:30:11.710261 16516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 006abbefcdac41d1915f9cc379f9c3a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "006abbefcdac41d1915f9cc379f9c3a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006abbefcdac41d1915f9cc379f9c3a7" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35479 } } }
I20260928 08:30:11.710456 16517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 006abbefcdac41d1915f9cc379f9c3a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 006abbefcdac41d1915f9cc379f9c3a7. Latest consensus state: current_term: 1 leader_uuid: "006abbefcdac41d1915f9cc379f9c3a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "006abbefcdac41d1915f9cc379f9c3a7" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35479 } } }
I20260928 08:30:11.710986 16516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 006abbefcdac41d1915f9cc379f9c3a7 [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:11.711007 16517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 006abbefcdac41d1915f9cc379f9c3a7 [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:11.712999 16523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:30:11.718616 16523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:30:11.728014 16523 catalog_manager.cc:1347] Generated new cluster ID: b6f0bc23ce50484aa055012a83ff3113
I20260928 08:30:11.728137 16523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:30:11.743520 16523 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:30:11.745095 16523 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:30:11.752748 16523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 006abbefcdac41d1915f9cc379f9c3a7: Generated new TSK 0
I20260928 08:30:11.753671 16523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.134.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
I20260928 08:30:11.804775 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:0
--local_ip_for_outbound_sockets=127.15.134.193
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:35479
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:41759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:12.187690 16536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:12.187988 16536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:12.188139 16536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:12.195773 16536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:12.195885 16536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:12.196094 16536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:30:12.206442 16536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:41759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:35479
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:12.207783 16536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:12.209435 16536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:12.224491 16541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:12.224464 16544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:12.224560 16542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:12.225418 16536 server_base.cc:1034] running on GCE node
I20260928 08:30:12.226120 16536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:12.227145 16536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:12.228448 16536 hybrid_clock.cc:648] HybridClock initialized: now 1790584212228298 us; error 76 us; skew 500 ppm
I20260928 08:30:12.228849 16536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:12 dist-test-slave-qp2n krb5kdc[16430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584212, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:12.231997 16536 init.cc:377] Logged in from keytab as kudu/127.15.134.193@KRBTEST.COM (short username kudu)
I20260928 08:30:12.233569 16536 webserver.cc:460] Webserver started at http://127.15.134.193:41921/ using document root <none> and password file <none>
I20260928 08:30:12.234256 16536 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:12.234398 16536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:12.234702 16536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:12.237715 16536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-0/data/instance:
uuid: "f01525232af04edcb021b033c07b1994"
format_stamp: "Formatted at 2026-09-28 08:30:12 on dist-test-slave-qp2n"
server_key: "b82cb4619953de858106b3e7838fa001"
server_key_iv: "78b9ff501d6e65251e2a55c08abb01c2"
server_key_version: "encryptionkey@0"
I20260928 08:30:12.238528 16536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance:
uuid: "f01525232af04edcb021b033c07b1994"
format_stamp: "Formatted at 2026-09-28 08:30:12 on dist-test-slave-qp2n"
server_key: "b82cb4619953de858106b3e7838fa001"
server_key_iv: "78b9ff501d6e65251e2a55c08abb01c2"
server_key_version: "encryptionkey@0"
I20260928 08:30:12.243732 16536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:30:12.247298 16551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:12.248792 16536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:30:12.248980 16536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "f01525232af04edcb021b033c07b1994"
format_stamp: "Formatted at 2026-09-28 08:30:12 on dist-test-slave-qp2n"
server_key: "b82cb4619953de858106b3e7838fa001"
server_key_iv: "78b9ff501d6e65251e2a55c08abb01c2"
server_key_version: "encryptionkey@0"
I20260928 08:30:12.249189 16536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:12.275207 16536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:12.279717 16536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:12.280040 16536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:12.281122 16536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:30:12.282557 16536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:12.282661 16536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:12.282809 16536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:12.282886 16536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:12.318148 16536 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:35635
I20260928 08:30:12.318197 16663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:35635 every 8 connection(s)
I20260928 08:30:12.319449 16536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:30:12.326982 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 16536
I20260928 08:30:12.327203 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance
I20260928 08:30:12.327800 15899 external_mini_cluster.cc:1428] Setting key 92069e4bb379f4afab2c99cda9a58a2b
Sep 28 08:30:12 dist-test-slave-qp2n krb5kdc[16430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584212, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:12.341179 16666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:12.322184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:52613 (local address 127.15.134.254:35479)
0928 08:30:12.322621 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:12.322625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:12.323710 (+ 1085us) server_negotiation.cc:407] Connection header received
0928 08:30:12.325640 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:12.325646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:12.325745 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:12.325871 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:12.327491 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:12.328245 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:12.330137 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:12.330519 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:12.330657 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:12.334402 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:12.334409 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:12.334413 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:12.334451 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:12.337301 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:12.338411 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:12.338417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:12.338419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:12.338571 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:12.339149 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:12.339153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:12.339155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:12.339511 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:12.339654 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:12.340738 (+ 1084us) server_negotiation.cc:299] Negotiation successful
0928 08:30:12.340930 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":219,"threads_started":1}
I20260928 08:30:12.342878 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:12.322771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:35479 (local address 127.15.134.193:52613)
0928 08:30:12.323554 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:12.323592 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:12.325362 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:12.326114 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:12.326123 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:12.326638 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:12.327306 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:12.327344 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:12.328389 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:12.328395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:12.329993 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:12.330006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:12.330750 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:12.330758 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:12.330997 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:12.331686 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:12.331714 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:12.334105 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:12.337450 (+ 3345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:12.337462 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:12.337482 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:12.338156 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:12.338704 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:12.338712 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:12.338718 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:12.338983 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:12.339661 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:12.339673 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:12.340193 (+ 520us) client_negotiation.cc:769] Sending connection context
0928 08:30:12.340548 (+ 355us) client_negotiation.cc:240] Negotiation successful
0928 08:30:12.341223 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":267,"threads_started":1}
I20260928 08:30:12.344718 16664 heartbeater.cc:347] Connected to a master server at 127.15.134.254:35479
WARNING: no policy specified for kudu/127.15.134.194@KRBTEST.COM; defaulting to no policy
I20260928 08:30:12.345096 16664 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:12.345866 16664 heartbeater.cc:511] Master 127.15.134.254:35479 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.194@KRBTEST.COM" created.
I20260928 08:30:12.348305 16475 ts_manager.cc:194] Registered new tserver with Master: f01525232af04edcb021b033c07b1994 (127.15.134.193:35635)
I20260928 08:30:12.350569 16475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.193@KRBTEST.COM'} at 127.15.134.193:52613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
I20260928 08:30:12.372114 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:0
--local_ip_for_outbound_sockets=127.15.134.194
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:35479
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:41759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:12.725502 16670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:12.725832 16670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:12.726009 16670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:12.733127 16670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:12.733279 16670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:12.733464 16670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:30:12.743469 16670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:41759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:35479
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:12.744906 16670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:12.746280 16670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:12.759967 16676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:12.759930 16678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:12.760236 16675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:12.760850 16670 server_base.cc:1034] running on GCE node
I20260928 08:30:12.761544 16670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:12.762717 16670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:12.763991 16670 hybrid_clock.cc:648] HybridClock initialized: now 1790584212763884 us; error 69 us; skew 500 ppm
I20260928 08:30:12.764333 16670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:12 dist-test-slave-qp2n krb5kdc[16430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584212, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:12.767630 16670 init.cc:377] Logged in from keytab as kudu/127.15.134.194@KRBTEST.COM (short username kudu)
I20260928 08:30:12.769309 16670 webserver.cc:460] Webserver started at http://127.15.134.194:34321/ using document root <none> and password file <none>
I20260928 08:30:12.770057 16670 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:12.770205 16670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:12.770530 16670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:12.773754 16670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-1/data/instance:
uuid: "05309289f5c74a58bd7aa41278071007"
format_stamp: "Formatted at 2026-09-28 08:30:12 on dist-test-slave-qp2n"
server_key: "1fd142fd5e7a058d9ce6f7965fa4f72d"
server_key_iv: "00c9b105d4355576f8116c23ac093205"
server_key_version: "encryptionkey@0"
I20260928 08:30:12.774583 16670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance:
uuid: "05309289f5c74a58bd7aa41278071007"
format_stamp: "Formatted at 2026-09-28 08:30:12 on dist-test-slave-qp2n"
server_key: "1fd142fd5e7a058d9ce6f7965fa4f72d"
server_key_iv: "00c9b105d4355576f8116c23ac093205"
server_key_version: "encryptionkey@0"
I20260928 08:30:12.779860 16670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:30:12.783437 16685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:12.784893 16670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:30:12.785089 16670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "05309289f5c74a58bd7aa41278071007"
format_stamp: "Formatted at 2026-09-28 08:30:12 on dist-test-slave-qp2n"
server_key: "1fd142fd5e7a058d9ce6f7965fa4f72d"
server_key_iv: "00c9b105d4355576f8116c23ac093205"
server_key_version: "encryptionkey@0"
I20260928 08:30:12.785290 16670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:12.823293 16670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:12.827517 16670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:12.827816 16670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:12.828847 16670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:30:12.830159 16670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:12.830272 16670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:12.830427 16670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:12.830502 16670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:12.864457 16670 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:45315
I20260928 08:30:12.864547 16797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:45315 every 8 connection(s)
I20260928 08:30:12.865815 16670 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:30:12.871132 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 16670
I20260928 08:30:12.871332 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance
I20260928 08:30:12.871760 15899 external_mini_cluster.cc:1428] Setting key 35fb68d774502fa7b6ccddbc758edd07
Sep 28 08:30:12 dist-test-slave-qp2n krb5kdc[16430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584212, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:12.885936 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:12.868344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:56145 (local address 127.15.134.254:35479)
0928 08:30:12.868827 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:12.868832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:12.869809 (+ 977us) server_negotiation.cc:407] Connection header received
0928 08:30:12.871813 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:12.871818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:12.871913 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:12.872078 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:12.873918 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:12.874634 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:12.876166 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:12.876583 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:12.876771 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:12.880177 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:12.880182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:12.880186 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:12.880217 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:12.882807 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:12.883620 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:12.883627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:12.883631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:12.883767 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:12.884197 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:12.884204 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:12.884207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:12.884474 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:12.884649 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:12.885332 (+ 683us) server_negotiation.cc:299] Negotiation successful
0928 08:30:12.885564 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":247,"threads_started":1}
I20260928 08:30:12.886963 16801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:12.868985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:35479 (local address 127.15.134.194:56145)
0928 08:30:12.869653 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:12.869691 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:12.871536 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:12.872320 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:12.872328 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:12.873011 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:12.873744 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:12.873764 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:12.874788 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:12.874799 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:12.876018 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:12.876029 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:12.876860 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:12.876866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:12.877102 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:12.877816 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:12.877841 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:12.879948 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:12.882959 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:12.882968 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:12.882986 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:12.883488 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:12.883889 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:12.883894 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:12.883897 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:12.884089 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:12.884602 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:12.884630 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:12.884983 (+ 353us) client_negotiation.cc:769] Sending connection context
0928 08:30:12.885248 (+ 265us) client_negotiation.cc:240] Negotiation successful
0928 08:30:12.885770 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":275,"threads_started":1}
WARNING: no policy specified for kudu/127.15.134.195@KRBTEST.COM; defaulting to no policy
I20260928 08:30:12.888548 16798 heartbeater.cc:347] Connected to a master server at 127.15.134.254:35479
I20260928 08:30:12.888898 16798 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:12.889725 16798 heartbeater.cc:511] Master 127.15.134.254:35479 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.195@KRBTEST.COM" created.
I20260928 08:30:12.891474 16475 ts_manager.cc:194] Registered new tserver with Master: 05309289f5c74a58bd7aa41278071007 (127.15.134.194:45315)
I20260928 08:30:12.892292 16475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.194@KRBTEST.COM'} at 127.15.134.194:56145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
I20260928 08:30:12.913368 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:0
--local_ip_for_outbound_sockets=127.15.134.195
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:35479
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:41759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:13.267935 16804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:13.268232 16804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:13.268407 16804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:13.275923 16804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:13.276057 16804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:13.276262 16804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:30:13.286612 16804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:41759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:35479
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:13.288038 16804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:13.289521 16804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:13.303998 16809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:13.304031 16812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:13.304087 16810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:13.305135 16804 server_base.cc:1034] running on GCE node
I20260928 08:30:13.305842 16804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:13.306882 16804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:13.308104 16804 hybrid_clock.cc:648] HybridClock initialized: now 1790584213308059 us; error 41 us; skew 500 ppm
I20260928 08:30:13.308473 16804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:13 dist-test-slave-qp2n krb5kdc[16430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584213, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:13.311976 16804 init.cc:377] Logged in from keytab as kudu/127.15.134.195@KRBTEST.COM (short username kudu)
I20260928 08:30:13.313745 16804 webserver.cc:460] Webserver started at http://127.15.134.195:39993/ using document root <none> and password file <none>
I20260928 08:30:13.314471 16804 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:13.314574 16804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:13.314836 16804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:13.317903 16804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-2/data/instance:
uuid: "1dd9585a0a294e1b9d3f2dc2597b5e18"
format_stamp: "Formatted at 2026-09-28 08:30:13 on dist-test-slave-qp2n"
server_key: "8eebc90e5f50392412a51c7193f3e2be"
server_key_iv: "0620b23839657d50ca148d85aabf2a33"
server_key_version: "encryptionkey@0"
I20260928 08:30:13.318743 16804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance:
uuid: "1dd9585a0a294e1b9d3f2dc2597b5e18"
format_stamp: "Formatted at 2026-09-28 08:30:13 on dist-test-slave-qp2n"
server_key: "8eebc90e5f50392412a51c7193f3e2be"
server_key_iv: "0620b23839657d50ca148d85aabf2a33"
server_key_version: "encryptionkey@0"
I20260928 08:30:13.324015 16804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260928 08:30:13.327577 16819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:13.329064 16804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:30:13.329216 16804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "1dd9585a0a294e1b9d3f2dc2597b5e18"
format_stamp: "Formatted at 2026-09-28 08:30:13 on dist-test-slave-qp2n"
server_key: "8eebc90e5f50392412a51c7193f3e2be"
server_key_iv: "0620b23839657d50ca148d85aabf2a33"
server_key_version: "encryptionkey@0"
I20260928 08:30:13.329401 16804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:13.353508 16804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:13.354408 16664 heartbeater.cc:503] Master 127.15.134.254:35479 was elected leader, sending a full tablet report...
I20260928 08:30:13.358681 16804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:13.359001 16804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:13.360078 16804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:30:13.361598 16804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:13.361709 16804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:13.361876 16804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:13.361959 16804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:13.400111 16804 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:46189
I20260928 08:30:13.400158 16931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:46189 every 8 connection(s)
I20260928 08:30:13.401737 16804 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:30:13.411870 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 16804
I20260928 08:30:13.412096 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance
I20260928 08:30:13.412549 15899 external_mini_cluster.cc:1428] Setting key a4c1e324757a130e388f365bb9d9c894
Sep 28 08:30:13 dist-test-slave-qp2n krb5kdc[16430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584213, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:13.424885 16934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:13.404485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:39927 (local address 127.15.134.254:35479)
0928 08:30:13.405092 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:13.405099 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:13.406221 (+ 1122us) server_negotiation.cc:407] Connection header received
0928 08:30:13.408148 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:13.408154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:13.408235 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:13.408359 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:13.410173 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.411148 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.413021 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.413628 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.413764 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:13.418256 (+ 4492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:13.418262 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:13.418267 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:13.418313 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:13.421346 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:13.422180 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:13.422185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:13.422188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:13.422338 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:13.422903 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:13.422907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:13.422909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:13.423207 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:13.423354 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:13.424277 (+ 923us) server_negotiation.cc:299] Negotiation successful
0928 08:30:13.424487 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":318,"threads_started":1}
I20260928 08:30:13.425889 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:13.405320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:35479 (local address 127.15.134.195:39927)
0928 08:30:13.406036 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:13.406081 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:13.407845 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:13.408725 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:13.408735 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:13.409279 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:13.409974 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:13.409995 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.411307 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.411311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:13.412867 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:13.412882 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.413888 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.413896 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:13.414251 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:13.415086 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:13.415110 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:13.418017 (+ 2907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:13.421519 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:13.421528 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:13.421544 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:13.422028 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:13.422449 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:13.422452 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:13.422454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:13.422754 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:13.423445 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:13.423453 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:13.423842 (+ 389us) client_negotiation.cc:769] Sending connection context
0928 08:30:13.424107 (+ 265us) client_negotiation.cc:240] Negotiation successful
0928 08:30:13.424557 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":279,"threads_started":1}
I20260928 08:30:13.427685 16932 heartbeater.cc:347] Connected to a master server at 127.15.134.254:35479
I20260928 08:30:13.428028 16932 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:13.429044 16932 heartbeater.cc:511] Master 127.15.134.254:35479 requested a full tablet report, sending...
I20260928 08:30:13.430632 16475 ts_manager.cc:194] Registered new tserver with Master: 1dd9585a0a294e1b9d3f2dc2597b5e18 (127.15.134.195:46189)
I20260928 08:30:13.431509 16475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.195@KRBTEST.COM'} at 127.15.134.195:39927
I20260928 08:30:13.442155 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:30:13.463547 16934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:13.449207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59590 (local address 127.15.134.254:35479)
0928 08:30:13.449413 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:13.449418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:13.450026 (+ 608us) server_negotiation.cc:407] Connection header received
0928 08:30:13.450184 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:13.450188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:13.450276 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:13.450376 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:13.452188 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.453067 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.455130 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.455545 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.455652 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:13.458213 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:13.458219 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:13.458222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:13.458259 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:13.461029 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:13.461749 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:13.461753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:13.461755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:13.461890 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:13.462243 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:13.462246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:13.462248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:13.462561 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:13.462747 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:13.463139 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 08:30:13.463305 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 08:30:13.470855 16475 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:30:13.478044 16475 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:30:13.504218 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:13.492763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51014 (local address 127.15.134.195:46189)
0928 08:30:13.493728 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:13.493738 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:13.494128 (+ 390us) server_negotiation.cc:407] Connection header received
0928 08:30:13.494234 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:13.494241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:13.494500 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:13.494690 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:30:13.496048 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.498704 (+ 2656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.501356 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.502741 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.502943 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:13.503131 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:13.503613 (+ 482us) server_negotiation.cc:299] Negotiation successful
0928 08:30:13.503808 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":620,"threads_started":1}
I20260928 08:30:13.504369 16945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:13.492150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:46189 (local address 127.0.0.1:51014)
0928 08:30:13.493846 (+ 1696us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:13.493867 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:13.493997 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:13.494964 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:13.494973 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:13.495014 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:30:13.495472 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:13.495484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.498873 (+ 3389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.498880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:13.501187 (+ 2307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:13.501201 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.502959 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.502967 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:13.503184 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:13.503238 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 08:30:13.503561 (+ 323us) client_negotiation.cc:240] Negotiation successful
0928 08:30:13.503799 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1553,"mutex_wait_us":173,"spinlock_wait_cycles":1792,"thread_start_us":1420,"threads_started":1}
I20260928 08:30:13.505092 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:13.494634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60414 (local address 127.15.134.193:35635)
0928 08:30:13.495571 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:13.495581 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:13.496236 (+ 655us) server_negotiation.cc:407] Connection header received
0928 08:30:13.496544 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:13.496554 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:13.496862 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:13.497071 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:30:13.497774 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.498941 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.501904 (+ 2963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.503414 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.503607 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:13.503750 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:13.504482 (+ 732us) server_negotiation.cc:299] Negotiation successful
0928 08:30:13.504684 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":751,"thread_start_us":543,"threads_started":1}
I20260928 08:30:13.505525 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:13.492740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53460 (local address 127.15.134.194:45315)
0928 08:30:13.494155 (+ 1415us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:13.494163 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:13.494186 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:30:13.494273 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:13.494279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:13.494500 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:13.494690 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:30:13.496512 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.497959 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.501682 (+ 3723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.502926 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.503073 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:13.503253 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:13.504999 (+ 1746us) server_negotiation.cc:299] Negotiation successful
0928 08:30:13.505181 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1241,"thread_start_us":1126,"threads_started":1}
I20260928 08:30:13.505617 16949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:13.494019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:35635 (local address 127.0.0.1:60414)
0928 08:30:13.496109 (+ 2090us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:13.496142 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:13.496339 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:13.497090 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:13.497095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:13.497126 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:30:13.497578 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:13.497590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.499155 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.499167 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:13.501727 (+ 2560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:13.501743 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.503909 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.503916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:13.504088 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:13.504116 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 08:30:13.504210 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 08:30:13.504335 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1944,"mutex_wait_us":1,"thread_start_us":1745,"threads_started":1}
I20260928 08:30:13.506026 16946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:13.492137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:45315 (local address 127.0.0.1:53460)
0928 08:30:13.493005 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:13.493053 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:13.493255 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:13.494964 (+ 1709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:13.494973 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:13.495021 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:30:13.496329 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:13.496339 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.498127 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.498134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:13.501471 (+ 3337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:13.501488 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.503391 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.503398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:13.503604 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:13.504806 (+ 1202us) client_negotiation.cc:769] Sending connection context
0928 08:30:13.505462 (+ 656us) client_negotiation.cc:240] Negotiation successful
0928 08:30:13.505601 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":653,"spinlock_wait_cycles":9344,"thread_start_us":815,"threads_started":1}
I20260928 08:30:13.508802 16867 tablet_service.cc:1467] Processing CreateTablet for tablet 473f4de9c0084ddfb60394ab0f6a024d (DEFAULT_TABLE table=test-table [id=3343c17a949141d085358a5c836cc0c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:13.509426 16733 tablet_service.cc:1467] Processing CreateTablet for tablet 473f4de9c0084ddfb60394ab0f6a024d (DEFAULT_TABLE table=test-table [id=3343c17a949141d085358a5c836cc0c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:13.509426 16599 tablet_service.cc:1467] Processing CreateTablet for tablet 473f4de9c0084ddfb60394ab0f6a024d (DEFAULT_TABLE table=test-table [id=3343c17a949141d085358a5c836cc0c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:13.510536 16867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 473f4de9c0084ddfb60394ab0f6a024d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:13.510783 16599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 473f4de9c0084ddfb60394ab0f6a024d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:13.510794 16733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 473f4de9c0084ddfb60394ab0f6a024d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:13.523454 16951 tablet_bootstrap.cc:492] T 473f4de9c0084ddfb60394ab0f6a024d P 1dd9585a0a294e1b9d3f2dc2597b5e18: Bootstrap starting.
I20260928 08:30:13.525108 16952 tablet_bootstrap.cc:492] T 473f4de9c0084ddfb60394ab0f6a024d P 05309289f5c74a58bd7aa41278071007: Bootstrap starting.
I20260928 08:30:13.525893 16953 tablet_bootstrap.cc:492] T 473f4de9c0084ddfb60394ab0f6a024d P f01525232af04edcb021b033c07b1994: Bootstrap starting.
I20260928 08:30:13.527506 16951 tablet_bootstrap.cc:654] T 473f4de9c0084ddfb60394ab0f6a024d P 1dd9585a0a294e1b9d3f2dc2597b5e18: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:13.528805 16951 log.cc:824] T 473f4de9c0084ddfb60394ab0f6a024d P 1dd9585a0a294e1b9d3f2dc2597b5e18: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:13.529508 16953 tablet_bootstrap.cc:654] T 473f4de9c0084ddfb60394ab0f6a024d P f01525232af04edcb021b033c07b1994: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:13.530519 16952 tablet_bootstrap.cc:654] T 473f4de9c0084ddfb60394ab0f6a024d P 05309289f5c74a58bd7aa41278071007: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:13.530624 16953 log.cc:824] T 473f4de9c0084ddfb60394ab0f6a024d P f01525232af04edcb021b033c07b1994: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:13.531739 16952 log.cc:824] T 473f4de9c0084ddfb60394ab0f6a024d P 05309289f5c74a58bd7aa41278071007: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:13.532630 16951 tablet_bootstrap.cc:492] T 473f4de9c0084ddfb60394ab0f6a024d P 1dd9585a0a294e1b9d3f2dc2597b5e18: No bootstrap required, opened a new log
I20260928 08:30:13.532945 16951 ts_tablet_manager.cc:1397] T 473f4de9c0084ddfb60394ab0f6a024d P 1dd9585a0a294e1b9d3f2dc2597b5e18: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 08:30:13.534732 16953 tablet_bootstrap.cc:492] T 473f4de9c0084ddfb60394ab0f6a024d P f01525232af04edcb021b033c07b1994: No bootstrap required, opened a new log
I20260928 08:30:13.535028 16953 ts_tablet_manager.cc:1397] T 473f4de9c0084ddfb60394ab0f6a024d P f01525232af04edcb021b033c07b1994: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260928 08:30:13.535164 16952 tablet_bootstrap.cc:492] T 473f4de9c0084ddfb60394ab0f6a024d P 05309289f5c74a58bd7aa41278071007: No bootstrap required, opened a new log
I20260928 08:30:13.535432 16952 ts_tablet_manager.cc:1397] T 473f4de9c0084ddfb60394ab0f6a024d P 05309289f5c74a58bd7aa41278071007: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260928 08:30:13.543689 16953 raft_consensus.cc:348] T 473f4de9c0084ddfb60394ab0f6a024d P f01525232af04edcb021b033c07b1994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05309289f5c74a58bd7aa41278071007" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 45315 } } peers { permanent_uuid: "1dd9585a0a294e1b9d3f2dc2597b5e18" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 46189 } } peers { permanent_uuid: "f01525232af04edcb021b033c07b1994" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 35635 } }
I20260928 08:30:13.543931 16951 raft_consensus.cc:348] T 473f4de9c0084ddfb60394ab0f6a024d P 1dd9585a0a294e1b9d3f2dc2597b5e18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05309289f5c74a58bd7aa41278071007" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 45315 } } peers { permanent_uuid: "1dd9585a0a294e1b9d3f2dc2597b5e18" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 46189 } } peers { permanent_uuid: "f01525232af04edcb021b033c07b1994" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 35635 } }
I20260928 08:30:13.544405 16951 raft_consensus.cc:374] T 473f4de9c0084ddfb60394ab0f6a024d P 1dd9585a0a294e1b9d3f2dc2597b5e18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:13.544433 16953 raft_consensus.cc:374] T 473f4de9c0084ddfb60394ab0f6a024d P f01525232af04edcb021b033c07b1994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:13.544543 16953 raft_consensus.cc:729] T 473f4de9c0084ddfb60394ab0f6a024d P f01525232af04edcb021b033c07b1994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f01525232af04edcb021b033c07b1994, State: Initialized, Role: FOLLOWER
I20260928 08:30:13.544708 16951 raft_consensus.cc:729] T 473f4de9c0084ddfb60394ab0f6a024d P 1dd9585a0a294e1b9d3f2dc2597b5e18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dd9585a0a294e1b9d3f2dc2597b5e18, State: Initialized, Role: FOLLOWER
I20260928 08:30:13.545347 16953 consensus_queue.cc:260] T 473f4de9c0084ddfb60394ab0f6a024d P f01525232af04edcb021b033c07b1994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05309289f5c74a58bd7aa41278071007" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 45315 } } peers { permanent_uuid: "1dd9585a0a294e1b9d3f2dc2597b5e18" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 46189 } } peers { permanent_uuid: "f01525232af04edcb021b033c07b1994" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 35635 } }
I20260928 08:30:13.545413 16951 consensus_queue.cc:260] T 473f4de9c0084ddfb60394ab0f6a024d P 1dd9585a0a294e1b9d3f2dc2597b5e18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05309289f5c74a58bd7aa41278071007" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 45315 } } peers { permanent_uuid: "1dd9585a0a294e1b9d3f2dc2597b5e18" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 46189 } } peers { permanent_uuid: "f01525232af04edcb021b033c07b1994" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 35635 } }
I20260928 08:30:13.547793 16952 raft_consensus.cc:348] T 473f4de9c0084ddfb60394ab0f6a024d P 05309289f5c74a58bd7aa41278071007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05309289f5c74a58bd7aa41278071007" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 45315 } } peers { permanent_uuid: "1dd9585a0a294e1b9d3f2dc2597b5e18" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 46189 } } peers { permanent_uuid: "f01525232af04edcb021b033c07b1994" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 35635 } }
I20260928 08:30:13.548456 16952 raft_consensus.cc:374] T 473f4de9c0084ddfb60394ab0f6a024d P 05309289f5c74a58bd7aa41278071007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:13.548643 16952 raft_consensus.cc:729] T 473f4de9c0084ddfb60394ab0f6a024d P 05309289f5c74a58bd7aa41278071007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05309289f5c74a58bd7aa41278071007, State: Initialized, Role: FOLLOWER
I20260928 08:30:13.549803 16952 consensus_queue.cc:260] T 473f4de9c0084ddfb60394ab0f6a024d P 05309289f5c74a58bd7aa41278071007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05309289f5c74a58bd7aa41278071007" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 45315 } } peers { permanent_uuid: "1dd9585a0a294e1b9d3f2dc2597b5e18" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 46189 } } peers { permanent_uuid: "f01525232af04edcb021b033c07b1994" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 35635 } }
I20260928 08:30:13.552927 16953 ts_tablet_manager.cc:1428] T 473f4de9c0084ddfb60394ab0f6a024d P f01525232af04edcb021b033c07b1994: Time spent starting tablet: real 0.018s user 0.012s sys 0.007s
I20260928 08:30:13.553925 16951 ts_tablet_manager.cc:1428] T 473f4de9c0084ddfb60394ab0f6a024d P 1dd9585a0a294e1b9d3f2dc2597b5e18: Time spent starting tablet: real 0.021s user 0.019s sys 0.000s
I20260928 08:30:13.554448 16932 heartbeater.cc:503] Master 127.15.134.254:35479 was elected leader, sending a full tablet report...
I20260928 08:30:13.555295 16798 heartbeater.cc:503] Master 127.15.134.254:35479 was elected leader, sending a full tablet report...
I20260928 08:30:13.556797 16952 ts_tablet_manager.cc:1428] T 473f4de9c0084ddfb60394ab0f6a024d P 05309289f5c74a58bd7aa41278071007: Time spent starting tablet: real 0.021s user 0.009s sys 0.012s
W20260928 08:30:13.572182 16665 tablet.cc:2369] T 473f4de9c0084ddfb60394ab0f6a024d P f01525232af04edcb021b033c07b1994: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:30:13.618093 16799 tablet.cc:2369] T 473f4de9c0084ddfb60394ab0f6a024d P 05309289f5c74a58bd7aa41278071007: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:30:13.653523 16933 tablet.cc:2369] T 473f4de9c0084ddfb60394ab0f6a024d P 1dd9585a0a294e1b9d3f2dc2597b5e18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:30:13.884166 16957 raft_consensus.cc:482] T 473f4de9c0084ddfb60394ab0f6a024d P f01525232af04edcb021b033c07b1994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:30:13.884486 16957 raft_consensus.cc:504] T 473f4de9c0084ddfb60394ab0f6a024d P f01525232af04edcb021b033c07b1994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05309289f5c74a58bd7aa41278071007" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 45315 } } peers { permanent_uuid: "1dd9585a0a294e1b9d3f2dc2597b5e18" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 46189 } } peers { permanent_uuid: "f01525232af04edcb021b033c07b1994" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 35635 } }
I20260928 08:30:13.886529 16957 leader_election.cc:288] T 473f4de9c0084ddfb60394ab0f6a024d P f01525232af04edcb021b033c07b1994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05309289f5c74a58bd7aa41278071007 (127.15.134.194:45315), 1dd9585a0a294e1b9d3f2dc2597b5e18 (127.15.134.195:46189)
I20260928 08:30:13.896782 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:13.887038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:60483 (local address 127.15.134.194:45315)
0928 08:30:13.887269 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:13.887277 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:13.888488 (+ 1211us) server_negotiation.cc:407] Connection header received
0928 08:30:13.888917 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:13.888922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:13.888991 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:13.889103 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:30:13.890390 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.891527 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.893783 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.895434 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.895565 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:13.895624 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:13.896213 (+ 589us) server_negotiation.cc:299] Negotiation successful
0928 08:30:13.896390 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 08:30:13.897064 16961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:13.887413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:45315 (local address 127.15.134.193:60483)
0928 08:30:13.888339 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:13.888364 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:13.888586 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:13.889125 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:13.889131 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:13.889164 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:30:13.890139 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:13.890155 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.891679 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.891685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:13.893599 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:13.893615 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.895676 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.895684 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:13.896006 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:13.896049 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 08:30:13.896172 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 08:30:13.896319 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":25,"thread_start_us":501,"threads_started":1}
I20260928 08:30:13.897564 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:13.887242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:49561 (local address 127.15.134.195:46189)
0928 08:30:13.887414 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:13.887419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:13.889843 (+ 2424us) server_negotiation.cc:407] Connection header received
0928 08:30:13.890133 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:13.890139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:13.890245 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:13.890353 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:30:13.891095 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.892248 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.894879 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.896217 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.896526 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:13.896649 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:13.897091 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 08:30:13.897290 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 08:30:13.898381 16960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:13.887425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:46189 (local address 127.15.134.193:49561)
0928 08:30:13.889691 (+ 2266us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:13.889708 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:13.889910 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:13.890387 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:13.890392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:13.890419 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:30:13.890885 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:13.890898 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.892419 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.892423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:13.894666 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:13.894683 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.896604 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.896649 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:13.896861 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:13.896896 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 08:30:13.897751 (+ 855us) client_negotiation.cc:240] Negotiation successful
0928 08:30:13.897920 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2106,"mutex_wait_us":25,"thread_start_us":1629,"threads_started":1}
I20260928 08:30:13.898547 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "473f4de9c0084ddfb60394ab0f6a024d" candidate_uuid: "f01525232af04edcb021b033c07b1994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05309289f5c74a58bd7aa41278071007" is_pre_election: true
I20260928 08:30:13.899389 16753 raft_consensus.cc:2454] T 473f4de9c0084ddfb60394ab0f6a024d P 05309289f5c74a58bd7aa41278071007 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f01525232af04edcb021b033c07b1994 in term 0.
I20260928 08:30:13.899976 16887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "473f4de9c0084ddfb60394ab0f6a024d" candidate_uuid: "f01525232af04edcb021b033c07b1994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd9585a0a294e1b9d3f2dc2597b5e18" is_pre_election: true
I20260928 08:30:13.900496 16887 raft_consensus.cc:2454] T 473f4de9c0084ddfb60394ab0f6a024d P 1dd9585a0a294e1b9d3f2dc2597b5e18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f01525232af04edcb021b033c07b1994 in term 0.
I20260928 08:30:13.900583 16552 leader_election.cc:302] T 473f4de9c0084ddfb60394ab0f6a024d P f01525232af04edcb021b033c07b1994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05309289f5c74a58bd7aa41278071007, f01525232af04edcb021b033c07b1994; no voters:
I20260928 08:30:13.901157 16957 raft_consensus.cc:2781] T 473f4de9c0084ddfb60394ab0f6a024d P f01525232af04edcb021b033c07b1994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:30:13.901324 16957 raft_consensus.cc:482] T 473f4de9c0084ddfb60394ab0f6a024d P f01525232af04edcb021b033c07b1994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:30:13.901453 16957 raft_consensus.cc:3037] T 473f4de9c0084ddfb60394ab0f6a024d P f01525232af04edcb021b033c07b1994 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:13.904146 16957 raft_consensus.cc:504] T 473f4de9c0084ddfb60394ab0f6a024d P f01525232af04edcb021b033c07b1994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05309289f5c74a58bd7aa41278071007" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 45315 } } peers { permanent_uuid: "1dd9585a0a294e1b9d3f2dc2597b5e18" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 46189 } } peers { permanent_uuid: "f01525232af04edcb021b033c07b1994" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 35635 } }
I20260928 08:30:13.904980 16957 leader_election.cc:288] T 473f4de9c0084ddfb60394ab0f6a024d P f01525232af04edcb021b033c07b1994 [CANDIDATE]: Term 1 election: Requested vote from peers 05309289f5c74a58bd7aa41278071007 (127.15.134.194:45315), 1dd9585a0a294e1b9d3f2dc2597b5e18 (127.15.134.195:46189)
I20260928 08:30:13.905575 16887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "473f4de9c0084ddfb60394ab0f6a024d" candidate_uuid: "f01525232af04edcb021b033c07b1994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dd9585a0a294e1b9d3f2dc2597b5e18"
I20260928 08:30:13.905616 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "473f4de9c0084ddfb60394ab0f6a024d" candidate_uuid: "f01525232af04edcb021b033c07b1994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05309289f5c74a58bd7aa41278071007"
I20260928 08:30:13.905804 16887 raft_consensus.cc:3037] T 473f4de9c0084ddfb60394ab0f6a024d P 1dd9585a0a294e1b9d3f2dc2597b5e18 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:13.905871 16753 raft_consensus.cc:3037] T 473f4de9c0084ddfb60394ab0f6a024d P 05309289f5c74a58bd7aa41278071007 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:13.908459 16753 raft_consensus.cc:2454] T 473f4de9c0084ddfb60394ab0f6a024d P 05309289f5c74a58bd7aa41278071007 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f01525232af04edcb021b033c07b1994 in term 1.
I20260928 08:30:13.908949 16552 leader_election.cc:302] T 473f4de9c0084ddfb60394ab0f6a024d P f01525232af04edcb021b033c07b1994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05309289f5c74a58bd7aa41278071007, f01525232af04edcb021b033c07b1994; no voters:
I20260928 08:30:13.909300 16957 raft_consensus.cc:2781] T 473f4de9c0084ddfb60394ab0f6a024d P f01525232af04edcb021b033c07b1994 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:13.909724 16887 raft_consensus.cc:2454] T 473f4de9c0084ddfb60394ab0f6a024d P 1dd9585a0a294e1b9d3f2dc2597b5e18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f01525232af04edcb021b033c07b1994 in term 1.
I20260928 08:30:13.909842 16957 raft_consensus.cc:686] T 473f4de9c0084ddfb60394ab0f6a024d P f01525232af04edcb021b033c07b1994 [term 1 LEADER]: Becoming Leader. State: Replica: f01525232af04edcb021b033c07b1994, State: Running, Role: LEADER
I20260928 08:30:13.910400 16957 consensus_queue.cc:237] T 473f4de9c0084ddfb60394ab0f6a024d P f01525232af04edcb021b033c07b1994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05309289f5c74a58bd7aa41278071007" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 45315 } } peers { permanent_uuid: "1dd9585a0a294e1b9d3f2dc2597b5e18" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 46189 } } peers { permanent_uuid: "f01525232af04edcb021b033c07b1994" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 35635 } }
I20260928 08:30:13.915416 16473 catalog_manager.cc:5495] T 473f4de9c0084ddfb60394ab0f6a024d P f01525232af04edcb021b033c07b1994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f01525232af04edcb021b033c07b1994 (127.15.134.193). New cstate: current_term: 1 leader_uuid: "f01525232af04edcb021b033c07b1994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05309289f5c74a58bd7aa41278071007" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 45315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dd9585a0a294e1b9d3f2dc2597b5e18" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 46189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f01525232af04edcb021b033c07b1994" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 35635 } health_report { overall_health: HEALTHY } } }
Sep 28 08:30:13 dist-test-slave-qp2n krb5kdc[16430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.193@KRBTEST.COM
I20260928 08:30:13.968952 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:13.951809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60416 (local address 127.15.134.193:35635)
0928 08:30:13.952012 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:13.952017 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:13.952935 (+ 918us) server_negotiation.cc:407] Connection header received
0928 08:30:13.953176 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:13.953181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:13.953248 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:13.953375 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:13.955783 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.956526 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.957858 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:13.958278 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:13.958426 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:13.963101 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:13.963115 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:13.963132 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:13.963171 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:13.966278 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:13.966958 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:13.966965 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:13.966968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:13.967099 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:13.967433 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:13.967436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:13.967438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:13.967769 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:13.967988 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:13.968373 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 08:30:13.968579 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 08:30:13 dist-test-slave-qp2n krb5kdc[16430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584213, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 08:30:14 dist-test-slave-qp2n krb5kdc[16430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584213, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.134.193@KRBTEST.COM
I20260928 08:30:14.017956 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:14.002326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60426 (local address 127.15.134.193:35635)
0928 08:30:14.002594 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:14.002602 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:14.002876 (+ 274us) server_negotiation.cc:407] Connection header received
0928 08:30:14.003113 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:14.003117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:14.003186 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:14.003297 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:14.004827 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:14.005629 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:14.007063 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:14.007516 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:14.007662 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:14.012125 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:14.012131 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:14.012135 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:14.012176 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:14.015000 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:14.015739 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:14.015745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:14.015750 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:14.015901 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:14.016344 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:14.016350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:14.016353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:14.016685 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:14.016941 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:14.017267 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 08:30:14.017519 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
Sep 28 08:30:14 dist-test-slave-qp2n krb5kdc[16430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 08:30:14 dist-test-slave-qp2n krb5kdc[16430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.193@KRBTEST.COM
I20260928 08:30:14.065804 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:14.050358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60436 (local address 127.15.134.193:35635)
0928 08:30:14.050607 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:14.050616 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:14.050834 (+ 218us) server_negotiation.cc:407] Connection header received
0928 08:30:14.051031 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:14.051035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:14.051108 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:14.051199 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:14.052737 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:14.053517 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:14.055232 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:14.055646 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:14.055758 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:14.060165 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:14.060171 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:14.060174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:14.060211 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:14.062984 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:14.063851 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:14.063856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:14.063858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:14.063999 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:14.064405 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:14.064409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:14.064410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:14.064741 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:14.064948 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:14.065237 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 08:30:14.065500 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260928 08:30:14.067600 16579 tablet_service.cc:2908] Rejecting scan request for tablet 473f4de9c0084ddfb60394ab0f6a024d: Uninitialized: clean time has not yet been initialized
I20260928 08:30:14.070452 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 16536
I20260928 08:30:14.085130 16950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:14.071539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60446 (local address 127.15.134.193:35635)
0928 08:30:14.071791 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:14.071799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:14.072133 (+ 334us) server_negotiation.cc:407] Connection header received
0928 08:30:14.072413 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:14.072417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:14.072476 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:14.072596 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:14.073937 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:14.074685 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:14.076010 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:14.076469 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:14.076611 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:14.079070 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:14.079075 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:14.079079 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:14.079111 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:14.082058 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:14.082743 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:14.082757 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:14.082762 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:14.082907 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:14.083321 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:14.083334 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:14.083337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:14.083611 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:14.084018 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:14.084367 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 08:30:14.084661 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 08:30:14.085888 16659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:14.180121 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 16536
I20260928 08:30:14.199503 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 16670
Sep 28 08:30:14 dist-test-slave-qp2n krb5kdc[16430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.194@KRBTEST.COM
I20260928 08:30:14.215279 16948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:14.200506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53462 (local address 127.15.134.194:45315)
0928 08:30:14.200729 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:14.200736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:14.200826 (+ 90us) server_negotiation.cc:407] Connection header received
0928 08:30:14.201077 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:14.201082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:14.201145 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:14.201264 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:14.202503 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:14.203198 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:14.204480 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:14.204968 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:14.205104 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:14.209298 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:14.209313 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:14.209330 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:14.209362 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:14.212318 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:14.213108 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:14.213116 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:14.213118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:14.213236 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:14.213627 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:14.213636 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:14.213638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:14.214093 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:14.214400 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:14.214714 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 08:30:14.214947 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 08:30:14.215969 16793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:14.304047 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 16670
I20260928 08:30:14.321429 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 16804
Sep 28 08:30:14 dist-test-slave-qp2n krb5kdc[16430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.195@KRBTEST.COM
I20260928 08:30:14.336812 16947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:14.322206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51020 (local address 127.15.134.195:46189)
0928 08:30:14.322390 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:14.322397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:14.322528 (+ 131us) server_negotiation.cc:407] Connection header received
0928 08:30:14.322751 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:14.322755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:14.322852 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:14.322959 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:14.324186 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:14.325050 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:14.326349 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:14.326734 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:14.326880 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:14.331205 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:14.331221 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:14.331239 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:14.331278 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:14.333959 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:14.334664 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:14.334671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:14.334674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:14.334809 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:14.335174 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:14.335179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:14.335183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:14.335536 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:14.335835 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:14.336125 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 08:30:14.336344 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:30:14.337474 16927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:14.436806 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 16804
I20260928 08:30:14.454276 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 16443
I20260928 08:30:14.455510 16505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:14.570791 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 16443
2026-09-28T08:30:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4683 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:30:14 dist-test-slave-qp2n krb5kdc[16981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:30:14 dist-test-slave-qp2n krb5kdc[16981](info): set up 2 sockets
Sep 28 08:30:14 dist-test-slave-qp2n krb5kdc[16981](info): commencing operation
krb5kdc: starting...
W20260928 08:30:15.666507 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:30:15 dist-test-slave-qp2n krb5kdc[16981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:30:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:30:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:30:15.861541 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:41709
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:36491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:41709
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:16.196208 16994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:16.196534 16994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:16.196732 16994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:16.204414 16994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:30:16.204545 16994 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 08:30:16.204613 16994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:16.204771 16994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:16.204834 16994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:30:16.204922 16994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:30:16.215250 16994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:36491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:41709
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:41709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:16.216802 16994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:16.218323 16994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:16.230954 16999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:16.230932 17002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:16.230903 17000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:16.232372 16994 server_base.cc:1034] running on GCE node
I20260928 08:30:16.233342 16994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:16.234869 16994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:16.236155 16994 hybrid_clock.cc:648] HybridClock initialized: now 1790584216236089 us; error 54 us; skew 500 ppm
I20260928 08:30:16.236756 16994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:16 dist-test-slave-qp2n krb5kdc[16981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584216, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:16.240381 16994 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:30:16.242166 16994 webserver.cc:460] Webserver started at http://127.15.134.254:44009/ using document root <none> and password file <none>
I20260928 08:30:16.242925 16994 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:16.243019 16994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:16.243333 16994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:16.246717 16994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "c9fa1c14909246ba87b26d7ff8f71131"
format_stamp: "Formatted at 2026-09-28 08:30:16 on dist-test-slave-qp2n"
server_key: "a4c24e32b92331ca51c248bafb9f80fb"
server_key_iv: "85df93a07c7ee7ca94eca076feaa3448"
server_key_version: "encryptionkey@0"
I20260928 08:30:16.247525 16994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "c9fa1c14909246ba87b26d7ff8f71131"
format_stamp: "Formatted at 2026-09-28 08:30:16 on dist-test-slave-qp2n"
server_key: "a4c24e32b92331ca51c248bafb9f80fb"
server_key_iv: "85df93a07c7ee7ca94eca076feaa3448"
server_key_version: "encryptionkey@0"
I20260928 08:30:16.253556 16994 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 08:30:16.257520 17009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:16.259086 16994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:30:16.259276 16994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "c9fa1c14909246ba87b26d7ff8f71131"
format_stamp: "Formatted at 2026-09-28 08:30:16 on dist-test-slave-qp2n"
server_key: "a4c24e32b92331ca51c248bafb9f80fb"
server_key_iv: "85df93a07c7ee7ca94eca076feaa3448"
server_key_version: "encryptionkey@0"
I20260928 08:30:16.259593 16994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:16.273097 16994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:16.277550 16994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:16.277853 16994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:16.302304 16994 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:41709
I20260928 08:30:16.302325 17070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:41709 every 8 connection(s)
I20260928 08:30:16.303820 16994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:30:16.307946 17071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:16.309712 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 16994
I20260928 08:30:16.309978 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:30:16.310859 15899 external_mini_cluster.cc:1428] Setting key 8ee8641893091be07be86290d1b5aad1
I20260928 08:30:16.321447 17071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9fa1c14909246ba87b26d7ff8f71131: Bootstrap starting.
I20260928 08:30:16.325716 17071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9fa1c14909246ba87b26d7ff8f71131: Neither blocks nor log segments found. Creating new log.
Sep 28 08:30:16 dist-test-slave-qp2n krb5kdc[16981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:16.327005 17071 log.cc:824] T 00000000000000000000000000000000 P c9fa1c14909246ba87b26d7ff8f71131: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:16.330884 17071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9fa1c14909246ba87b26d7ff8f71131: No bootstrap required, opened a new log
I20260928 08:30:16.338182 17074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:16.313433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35166 (local address 127.15.134.254:41709)
0928 08:30:16.314298 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:16.314310 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:16.314357 (+ 47us) server_negotiation.cc:407] Connection header received
0928 08:30:16.316155 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:16.316178 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:16.316580 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:16.316988 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:16.318881 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:16.320332 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:16.322091 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:16.322862 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:16.323040 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:16.328591 (+ 5551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:16.328604 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:16.328659 (+ 55us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:16.328730 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:16.332916 (+ 4186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:16.333870 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:16.333879 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:16.333888 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:16.334111 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:16.334563 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:16.334568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:16.334571 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:16.335355 (+ 784us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:16.335658 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:16.336105 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 08:30:16.336650 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":425,"threads_started":1}
I20260928 08:30:16.340989 17071 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9fa1c14909246ba87b26d7ff8f71131 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fa1c14909246ba87b26d7ff8f71131" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41709 } }
I20260928 08:30:16.341432 17071 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9fa1c14909246ba87b26d7ff8f71131 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:16.341566 17071 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9fa1c14909246ba87b26d7ff8f71131 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9fa1c14909246ba87b26d7ff8f71131, State: Initialized, Role: FOLLOWER
I20260928 08:30:16.342147 17071 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9fa1c14909246ba87b26d7ff8f71131 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fa1c14909246ba87b26d7ff8f71131" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41709 } }
I20260928 08:30:16.342404 17071 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9fa1c14909246ba87b26d7ff8f71131 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:30:16.342548 17071 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9fa1c14909246ba87b26d7ff8f71131 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:30:16.342733 17071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9fa1c14909246ba87b26d7ff8f71131 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:16.345064 17071 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9fa1c14909246ba87b26d7ff8f71131 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fa1c14909246ba87b26d7ff8f71131" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41709 } }
I20260928 08:30:16.345603 17071 leader_election.cc:302] T 00000000000000000000000000000000 P c9fa1c14909246ba87b26d7ff8f71131 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9fa1c14909246ba87b26d7ff8f71131; no voters:
I20260928 08:30:16.346266 17071 leader_election.cc:288] T 00000000000000000000000000000000 P c9fa1c14909246ba87b26d7ff8f71131 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:30:16.346599 17076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9fa1c14909246ba87b26d7ff8f71131 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:16.347401 17076 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9fa1c14909246ba87b26d7ff8f71131 [term 1 LEADER]: Becoming Leader. State: Replica: c9fa1c14909246ba87b26d7ff8f71131, State: Running, Role: LEADER
I20260928 08:30:16.348042 17076 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9fa1c14909246ba87b26d7ff8f71131 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fa1c14909246ba87b26d7ff8f71131" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41709 } }
I20260928 08:30:16.348776 17071 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9fa1c14909246ba87b26d7ff8f71131 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:30:16.351423 17077 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9fa1c14909246ba87b26d7ff8f71131 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9fa1c14909246ba87b26d7ff8f71131" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fa1c14909246ba87b26d7ff8f71131" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41709 } } }
I20260928 08:30:16.352054 17077 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9fa1c14909246ba87b26d7ff8f71131 [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:16.352262 17078 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9fa1c14909246ba87b26d7ff8f71131 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9fa1c14909246ba87b26d7ff8f71131. Latest consensus state: current_term: 1 leader_uuid: "c9fa1c14909246ba87b26d7ff8f71131" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9fa1c14909246ba87b26d7ff8f71131" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41709 } } }
I20260928 08:30:16.352634 17078 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9fa1c14909246ba87b26d7ff8f71131 [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:16.355276 17085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:30:16.360548 17085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:30:16.370793 17085 catalog_manager.cc:1347] Generated new cluster ID: 422d3da26e334962a7222c0195c81df7
I20260928 08:30:16.370944 17085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:30:16.395442 17085 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:30:16.397087 17085 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:30:16.411163 17085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9fa1c14909246ba87b26d7ff8f71131: Generated new TSK 0
I20260928 08:30:16.412081 17085 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.134.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
I20260928 08:30:16.464432 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:0
--local_ip_for_outbound_sockets=127.15.134.193
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:41709
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:36491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:16.818859 17097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:16.819124 17097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:16.819221 17097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:16.826491 17097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:16.826601 17097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:16.826781 17097 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:30:16.826906 17097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:30:16.837148 17097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:36491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.134.254:41709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:16.838531 17097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:16.839908 17097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:16.853672 17103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:16.853703 17102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:16.854558 17097 server_base.cc:1034] running on GCE node
W20260928 08:30:16.854720 17105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:16.855450 17097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:16.856427 17097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:16.857649 17097 hybrid_clock.cc:648] HybridClock initialized: now 1790584216857579 us; error 55 us; skew 500 ppm
I20260928 08:30:16.858007 17097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:16 dist-test-slave-qp2n krb5kdc[16981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584216, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:16.861876 17097 init.cc:377] Logged in from keytab as kudu/127.15.134.193@KRBTEST.COM (short username kudu)
I20260928 08:30:16.863567 17097 webserver.cc:460] Webserver started at http://127.15.134.193:45551/ using document root <none> and password file <none>
I20260928 08:30:16.864353 17097 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:16.864442 17097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:16.864917 17097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:16.867892 17097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-0/data/instance:
uuid: "f40aabf45ef643bfb5ec1cf91fef2229"
format_stamp: "Formatted at 2026-09-28 08:30:16 on dist-test-slave-qp2n"
server_key: "b57ef4931a26e1f8b0c59cdfd1d00dec"
server_key_iv: "44bce8df43a4e23a4d15e45a762c1fa2"
server_key_version: "encryptionkey@0"
I20260928 08:30:16.868757 17097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance:
uuid: "f40aabf45ef643bfb5ec1cf91fef2229"
format_stamp: "Formatted at 2026-09-28 08:30:16 on dist-test-slave-qp2n"
server_key: "b57ef4931a26e1f8b0c59cdfd1d00dec"
server_key_iv: "44bce8df43a4e23a4d15e45a762c1fa2"
server_key_version: "encryptionkey@0"
I20260928 08:30:16.874035 17097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 08:30:16.877562 17112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:16.878937 17097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:30:16.879125 17097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "f40aabf45ef643bfb5ec1cf91fef2229"
format_stamp: "Formatted at 2026-09-28 08:30:16 on dist-test-slave-qp2n"
server_key: "b57ef4931a26e1f8b0c59cdfd1d00dec"
server_key_iv: "44bce8df43a4e23a4d15e45a762c1fa2"
server_key_version: "encryptionkey@0"
I20260928 08:30:16.879344 17097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:16.902901 17097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:16.907349 17097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:16.907661 17097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:16.910988 17097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:16.911105 17097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:16.911507 17097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:16.911588 17097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:30:16 dist-test-slave-qp2n krb5kdc[16981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584216, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:16.940297 17122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:16.912178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41709 (local address 127.15.134.193:48291)
0928 08:30:16.913136 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:16.913183 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:16.915373 (+ 2190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:16.916309 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:16.916322 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:16.917135 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:16.918407 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:16.918442 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:16.920097 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:16.920103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:16.923323 (+ 3220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:16.923352 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:16.924793 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:16.924807 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:16.925172 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:16.926162 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:16.926221 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:16.929839 (+ 3618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:16.934279 (+ 4440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:16.934293 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:16.934318 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:16.935146 (+ 828us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:16.935636 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:16.935641 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:16.935645 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:16.936058 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:16.936843 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:16.936851 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:16.937432 (+ 581us) client_negotiation.cc:769] Sending connection context
0928 08:30:16.937791 (+ 359us) client_negotiation.cc:240] Negotiation successful
0928 08:30:16.938627 (+ 836us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":412,"threads_started":1}
I20260928 08:30:16.941425 17120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:16.911488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:48291 (local address 127.15.134.254:41709)
0928 08:30:16.912196 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:16.912206 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:16.913313 (+ 1107us) server_negotiation.cc:407] Connection header received
0928 08:30:16.915736 (+ 2423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:16.915742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:16.915849 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:16.915981 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:16.918647 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:16.919865 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:16.923580 (+ 3715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:16.924161 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:16.924346 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:16.930181 (+ 5835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:16.930213 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:16.930219 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:16.930275 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:16.934080 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:16.935336 (+ 1256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:16.935342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:16.935346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:16.935499 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:16.936206 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:16.936211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:16.936215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:16.936677 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:16.936854 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:16.940987 (+ 4133us) server_negotiation.cc:299] Negotiation successful
0928 08:30:16.941174 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":447,"threads_started":1}
I20260928 08:30:16.963918 17205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:16.947464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41709 (local address 127.15.134.193:33925)
0928 08:30:16.948256 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:16.948291 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:16.948493 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:16.949084 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:16.949093 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:16.949904 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:16.950436 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:16.950455 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:16.951686 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:16.951692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:16.953786 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:16.953804 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:16.954691 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:16.954708 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:16.954886 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:16.955768 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:16.955787 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:16.956524 (+ 737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:16.960953 (+ 4429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:16.960957 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:16.960959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:16.961391 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:16.961822 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:16.961825 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:16.961828 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:16.961948 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:16.962615 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:16.962622 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:16.962895 (+ 273us) client_negotiation.cc:769] Sending connection context
0928 08:30:16.963049 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 08:30:16.963372 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":387,"threads_started":1}
I20260928 08:30:16.964062 17097 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:38189
I20260928 08:30:16.964164 17232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:38189 every 8 connection(s)
I20260928 08:30:16.964821 17120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:16.947479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:33925 (local address 127.15.134.254:41709)
0928 08:30:16.947706 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:16.947717 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:16.948411 (+ 694us) server_negotiation.cc:407] Connection header received
0928 08:30:16.948814 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:16.948819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:16.948890 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:16.948973 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:16.950656 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:16.951496 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:16.953947 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:16.954455 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:16.954605 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:16.956767 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:16.956779 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:16.956786 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:16.956826 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:16.960804 (+ 3978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:16.961518 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:16.961527 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:16.961533 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:16.961707 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:16.962057 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:16.962062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:16.962065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:16.962452 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:16.962643 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:16.964257 (+ 1614us) server_negotiation.cc:299] Negotiation successful
0928 08:30:16.964453 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:30:16.965549 17097 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:30:16.967924 17233 heartbeater.cc:347] Connected to a master server at 127.15.134.254:41709
I20260928 08:30:16.968215 17233 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:16.969122 17233 heartbeater.cc:511] Master 127.15.134.254:41709 requested a full tablet report, sending...
I20260928 08:30:16.971072 17026 ts_manager.cc:194] Registered new tserver with Master: f40aabf45ef643bfb5ec1cf91fef2229 (127.15.134.193:38189)
I20260928 08:30:16.972541 17026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.193@KRBTEST.COM'} at 127.15.134.193:48291
I20260928 08:30:16.974192 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 17097
I20260928 08:30:16.974438 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance
I20260928 08:30:16.975068 15899 external_mini_cluster.cc:1428] Setting key 9f54deb9300ccbd29aefb6f5fbfa27c6
WARNING: no policy specified for kudu/127.15.134.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
I20260928 08:30:17.016363 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:0
--local_ip_for_outbound_sockets=127.15.134.194
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:41709
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:36491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:17.355193 17237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:17.355557 17237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:17.355655 17237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:17.363005 17237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:17.363161 17237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:17.363358 17237 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:30:17.363493 17237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:30:17.373481 17237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:36491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.134.254:41709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:17.374848 17237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:17.376212 17237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:17.390486 17242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:17.390403 17245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:17.390431 17243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:17.391168 17237 server_base.cc:1034] running on GCE node
I20260928 08:30:17.391852 17237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:17.392792 17237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:17.394024 17237 hybrid_clock.cc:648] HybridClock initialized: now 1790584217393965 us; error 37 us; skew 500 ppm
I20260928 08:30:17.394369 17237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:17 dist-test-slave-qp2n krb5kdc[16981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584217, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:17.402050 17237 init.cc:377] Logged in from keytab as kudu/127.15.134.194@KRBTEST.COM (short username kudu)
I20260928 08:30:17.403687 17237 webserver.cc:460] Webserver started at http://127.15.134.194:36787/ using document root <none> and password file <none>
I20260928 08:30:17.404421 17237 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:17.404549 17237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:17.404898 17237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:17.407891 17237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-1/data/instance:
uuid: "ff685508c6c34fa78d511321ed9467ae"
format_stamp: "Formatted at 2026-09-28 08:30:17 on dist-test-slave-qp2n"
server_key: "534563d65e4bf13bda394e84c8148723"
server_key_iv: "645ef1f837f754cd4afafb5550154077"
server_key_version: "encryptionkey@0"
I20260928 08:30:17.408843 17237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance:
uuid: "ff685508c6c34fa78d511321ed9467ae"
format_stamp: "Formatted at 2026-09-28 08:30:17 on dist-test-slave-qp2n"
server_key: "534563d65e4bf13bda394e84c8148723"
server_key_iv: "645ef1f837f754cd4afafb5550154077"
server_key_version: "encryptionkey@0"
I20260928 08:30:17.414603 17237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:30:17.418268 17252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:17.419755 17237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:30:17.419947 17237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "ff685508c6c34fa78d511321ed9467ae"
format_stamp: "Formatted at 2026-09-28 08:30:17 on dist-test-slave-qp2n"
server_key: "534563d65e4bf13bda394e84c8148723"
server_key_iv: "645ef1f837f754cd4afafb5550154077"
server_key_version: "encryptionkey@0"
I20260928 08:30:17.420186 17237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:17.440125 17237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:17.444883 17237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:17.445156 17237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:17.447973 17237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:17.448091 17237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:17.448220 17237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:17.448401 17237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:30:17 dist-test-slave-qp2n krb5kdc[16981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584217, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:17.473264 17120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:17.448594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:55377 (local address 127.15.134.254:41709)
0928 08:30:17.448874 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:17.448881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:17.450938 (+ 2057us) server_negotiation.cc:407] Connection header received
0928 08:30:17.453319 (+ 2381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:17.453325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:17.453454 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:17.453569 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:17.456190 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:17.457362 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:17.459075 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:17.459520 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:17.459665 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:17.465444 (+ 5779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:17.465453 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:17.465461 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:17.465513 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:17.468549 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:17.469849 (+ 1300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:17.469854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:17.469857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:17.469984 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:17.470608 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:17.470617 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:17.470620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:17.471065 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:17.471244 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:17.472385 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0928 08:30:17.472753 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260928 08:30:17.474536 17261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:17.449676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41709 (local address 127.15.134.194:55377)
0928 08:30:17.450757 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:17.450799 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:17.452970 (+ 2171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:17.453938 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:17.453954 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:17.454890 (+ 936us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:17.455963 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:17.455996 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:17.457524 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:17.457536 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:17.458910 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:17.458923 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:17.459781 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:17.459788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:17.460023 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:17.460960 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:17.461009 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:17.465120 (+ 4111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:17.468759 (+ 3639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:17.468772 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:17.468804 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:17.469708 (+ 904us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:17.470121 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:17.470127 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:17.470131 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:17.470484 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:17.471193 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:17.471203 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:17.471839 (+ 636us) client_negotiation.cc:769] Sending connection context
0928 08:30:17.472214 (+ 375us) client_negotiation.cc:240] Negotiation successful
0928 08:30:17.472910 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"thread_start_us":536,"threads_started":1}
I20260928 08:30:17.500873 17120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:17.484320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:42241 (local address 127.15.134.254:41709)
0928 08:30:17.484580 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:17.484590 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:17.485047 (+ 457us) server_negotiation.cc:407] Connection header received
0928 08:30:17.485224 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:17.485233 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:17.485348 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:17.485504 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:17.486607 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:17.488148 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:17.490243 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:17.490676 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:17.490838 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:17.493289 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:17.493298 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:17.493305 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:17.493351 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:17.497444 (+ 4093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:17.498333 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:17.498342 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:17.498347 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:17.498550 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:17.499093 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:17.499100 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:17.499103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:17.499564 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:17.499725 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:17.500228 (+ 503us) server_negotiation.cc:299] Negotiation successful
0928 08:30:17.500466 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 08:30:17.501026 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:17.484223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41709 (local address 127.15.134.194:42241)
0928 08:30:17.484919 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:17.484941 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:17.485090 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:17.485564 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:17.485568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:17.486116 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:17.486435 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:17.486445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:17.488390 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:17.488395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:17.490054 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:17.490077 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:17.490895 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:17.490901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:17.491146 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:17.491988 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:17.492035 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:17.493012 (+ 977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:17.497617 (+ 4605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:17.497628 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:17.497636 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:17.498128 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:17.498677 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:17.498682 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:17.498686 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:17.498923 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:17.499790 (+ 867us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:17.499793 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:17.499965 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 08:30:17.500128 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 08:30:17.500493 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":360,"threads_started":1}
I20260928 08:30:17.514066 17237 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:33293
I20260928 08:30:17.514096 17371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:33293 every 8 connection(s)
I20260928 08:30:17.515304 17237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:30:17.517380 17372 heartbeater.cc:347] Connected to a master server at 127.15.134.254:41709
I20260928 08:30:17.517577 17372 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:17.518213 17372 heartbeater.cc:511] Master 127.15.134.254:41709 requested a full tablet report, sending...
I20260928 08:30:17.519430 17026 ts_manager.cc:194] Registered new tserver with Master: ff685508c6c34fa78d511321ed9467ae (127.15.134.194:33293)
I20260928 08:30:17.520254 17026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.194@KRBTEST.COM'} at 127.15.134.194:55377
I20260928 08:30:17.525226 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 17237
I20260928 08:30:17.525398 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance
I20260928 08:30:17.525777 15899 external_mini_cluster.cc:1428] Setting key 796f49fc7461db11f01364aee23ead09
WARNING: no policy specified for kudu/127.15.134.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
I20260928 08:30:17.567682 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:0
--local_ip_for_outbound_sockets=127.15.134.195
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:41709
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:36491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:17.901997 17376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:17.902307 17376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:17.902480 17376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:17.909946 17376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:17.910068 17376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:17.910238 17376 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:30:17.910375 17376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:30:17.920748 17376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:36491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.134.254:41709
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:17.922112 17376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:17.923512 17376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:17.937208 17382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:17.937285 17381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:17.937256 17384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:17.938158 17376 server_base.cc:1034] running on GCE node
I20260928 08:30:17.938802 17376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:17.939845 17376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:17.941058 17376 hybrid_clock.cc:648] HybridClock initialized: now 1790584217940988 us; error 46 us; skew 500 ppm
I20260928 08:30:17.941365 17376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:17 dist-test-slave-qp2n krb5kdc[16981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584217, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:17.944734 17376 init.cc:377] Logged in from keytab as kudu/127.15.134.195@KRBTEST.COM (short username kudu)
I20260928 08:30:17.946236 17376 webserver.cc:460] Webserver started at http://127.15.134.195:36501/ using document root <none> and password file <none>
I20260928 08:30:17.946909 17376 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:17.946998 17376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:17.947265 17376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:17.950424 17376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-2/data/instance:
uuid: "a2c8ba3451354e5882c303194b0ea691"
format_stamp: "Formatted at 2026-09-28 08:30:17 on dist-test-slave-qp2n"
server_key: "5f0fd0af2e1b5e6e8385ace9d4822ff5"
server_key_iv: "1791b8f4e6051e2e36d3a5700f020a19"
server_key_version: "encryptionkey@0"
I20260928 08:30:17.951184 17376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance:
uuid: "a2c8ba3451354e5882c303194b0ea691"
format_stamp: "Formatted at 2026-09-28 08:30:17 on dist-test-slave-qp2n"
server_key: "5f0fd0af2e1b5e6e8385ace9d4822ff5"
server_key_iv: "1791b8f4e6051e2e36d3a5700f020a19"
server_key_version: "encryptionkey@0"
I20260928 08:30:17.956564 17376 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 08:30:17.960440 17391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:17.961930 17376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:30:17.962110 17376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "a2c8ba3451354e5882c303194b0ea691"
format_stamp: "Formatted at 2026-09-28 08:30:17 on dist-test-slave-qp2n"
server_key: "5f0fd0af2e1b5e6e8385ace9d4822ff5"
server_key_iv: "1791b8f4e6051e2e36d3a5700f020a19"
server_key_version: "encryptionkey@0"
I20260928 08:30:17.962342 17376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:17.975644 17233 heartbeater.cc:503] Master 127.15.134.254:41709 was elected leader, sending a full tablet report...
I20260928 08:30:17.986884 17376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:17.991317 17376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:17.991613 17376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:17.994481 17376 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:17.994593 17376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:17.994755 17376 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:17.994859 17376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:30:18 dist-test-slave-qp2n krb5kdc[16981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584217, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:18.018007 17120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:17.995048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:55097 (local address 127.15.134.254:41709)
0928 08:30:17.995249 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:17.995257 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:17.996828 (+ 1571us) server_negotiation.cc:407] Connection header received
0928 08:30:17.998925 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:17.998931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:17.999032 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:17.999177 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:18.000692 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.001965 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.003615 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.004516 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.004713 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.009950 (+ 5237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:18.009956 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:18.009959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:18.009993 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:18.014003 (+ 4010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:18.014944 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:18.014952 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:18.014958 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:18.015142 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:18.015826 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:18.015831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:18.015834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:18.016165 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:18.016342 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:18.017402 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0928 08:30:18.017627 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:30:18.019439 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:17.995695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41709 (local address 127.15.134.195:55097)
0928 08:30:17.996593 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:17.996677 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:17.998610 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:17.999371 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:17.999380 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:17.999904 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:18.000484 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.000501 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.002142 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.002151 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.003464 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.003476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.004831 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.004844 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.005117 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.005935 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:18.005973 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:18.009649 (+ 3676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:18.014158 (+ 4509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:18.014170 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:18.014192 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:18.014799 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:18.015351 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:18.015360 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:18.015366 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:18.015680 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:18.016325 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:18.016333 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:18.016818 (+ 485us) client_negotiation.cc:769] Sending connection context
0928 08:30:18.017252 (+ 434us) client_negotiation.cc:240] Negotiation successful
0928 08:30:18.017793 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":392,"threads_started":1}
I20260928 08:30:18.041376 17120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.026205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:50171 (local address 127.15.134.254:41709)
0928 08:30:18.026380 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:18.026384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:18.027041 (+ 657us) server_negotiation.cc:407] Connection header received
0928 08:30:18.027248 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:18.027259 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:18.027319 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:18.027429 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:18.028418 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.029417 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.030794 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.031268 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.031428 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.033685 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:18.033693 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:18.033697 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:18.033735 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:18.037820 (+ 4085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:18.038893 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:18.038901 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:18.038907 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:18.039091 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:18.039595 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:18.039604 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:18.039608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:18.040067 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:18.040267 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:18.040904 (+ 637us) server_negotiation.cc:299] Negotiation successful
0928 08:30:18.041125 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:30:18.041780 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.026283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41709 (local address 127.15.134.195:50171)
0928 08:30:18.026934 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:18.026951 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:18.027109 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:18.027451 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:18.027456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:18.027954 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:18.028266 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.028275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.029568 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.029574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.030642 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.030655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.031472 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.031479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.031680 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.032353 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:18.032372 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:18.033450 (+ 1078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:18.038011 (+ 4561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:18.038025 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:18.038030 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:18.038745 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:18.039247 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:18.039254 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:18.039260 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:18.039445 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:18.040220 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:18.040225 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:18.040517 (+ 292us) client_negotiation.cc:769] Sending connection context
0928 08:30:18.040790 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 08:30:18.041188 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":367,"threads_started":1}
I20260928 08:30:18.050408 17376 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:45483
I20260928 08:30:18.050429 17510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:45483 every 8 connection(s)
I20260928 08:30:18.051934 17376 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:30:18.054539 17511 heartbeater.cc:347] Connected to a master server at 127.15.134.254:41709
I20260928 08:30:18.054777 17511 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:18.055429 17511 heartbeater.cc:511] Master 127.15.134.254:41709 requested a full tablet report, sending...
I20260928 08:30:18.056253 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 17376
I20260928 08:30:18.056452 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance
I20260928 08:30:18.057052 15899 external_mini_cluster.cc:1428] Setting key 7525fa8504317444a9af86c3fea805df
I20260928 08:30:18.057088 17026 ts_manager.cc:194] Registered new tserver with Master: a2c8ba3451354e5882c303194b0ea691 (127.15.134.195:45483)
I20260928 08:30:18.058243 17026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.195@KRBTEST.COM'} at 127.15.134.195:55097
I20260928 08:30:18.060230 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 08:30:18 dist-test-slave-qp2n krb5kdc[16981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584218, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 08:30:18 dist-test-slave-qp2n krb5kdc[16981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584218, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:18.109545 17120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.093544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33140 (local address 127.15.134.254:41709)
0928 08:30:18.093819 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:18.093826 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:18.094084 (+ 258us) server_negotiation.cc:407] Connection header received
0928 08:30:18.094349 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:18.094354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:18.094427 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:18.094544 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:18.096129 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.097014 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.098459 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.098964 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.099126 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.103637 (+ 4511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:18.103642 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:18.103645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:18.103688 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:18.106971 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:18.107591 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:18.107595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:18.107598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:18.107736 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:18.108095 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:18.108099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:18.108101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:18.108496 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:18.108805 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:18.108996 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 08:30:18.109275 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 08:30:18.114300 17026 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:30:18.121721 17026 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:30:18.156008 17529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.137294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56836 (local address 127.15.134.195:45483)
0928 08:30:18.145766 (+ 8472us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:18.145777 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:18.146975 (+ 1198us) server_negotiation.cc:407] Connection header received
0928 08:30:18.147107 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:18.147118 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:18.147530 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:18.148419 (+ 889us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:30:18.148815 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.150551 (+ 1736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.153797 (+ 3246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.154871 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.155092 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.155280 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:18.155493 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 08:30:18.155652 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8241,"thread_start_us":8324,"threads_started":1}
I20260928 08:30:18.156105 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.135867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:45483 (local address 127.0.0.1:56836)
0928 08:30:18.146518 (+ 10651us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:18.146540 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:18.146655 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:18.147790 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:18.147798 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:18.147829 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:30:18.148223 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.148239 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.150711 (+ 2472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.150717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.153617 (+ 2900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.153629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.155066 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.155074 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.155246 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.155291 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 08:30:18.155445 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 08:30:18.155660 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10501,"mutex_wait_us":231,"thread_start_us":8794,"threads_started":1}
I20260928 08:30:18.156858 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.136319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:33293 (local address 127.0.0.1:33890)
0928 08:30:18.147462 (+ 11143us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:18.147481 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:18.147614 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:18.148365 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:18.148374 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:18.148400 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:30:18.148933 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.148950 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.150668 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.150673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.154006 (+ 3333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.154018 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.156047 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.156058 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.156257 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.156283 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 08:30:18.156347 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 08:30:18.156470 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11022,"mutex_wait_us":273,"thread_start_us":9915,"threads_started":1}
I20260928 08:30:18.157203 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.136809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33890 (local address 127.15.134.194:33293)
0928 08:30:18.146146 (+ 9337us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:18.146158 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:18.147769 (+ 1611us) server_negotiation.cc:407] Connection header received
0928 08:30:18.147864 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:18.147870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:18.148142 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:18.148358 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:30:18.149104 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.150524 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.154201 (+ 3677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.155548 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.155721 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.155886 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:18.156754 (+ 868us) server_negotiation.cc:299] Negotiation successful
0928 08:30:18.156909 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9086,"spinlock_wait_cycles":29440,"thread_start_us":9069,"threads_started":1}
I20260928 08:30:18.159782 17441 tablet_service.cc:1467] Processing CreateTablet for tablet 705d224a4cf64e0abd32fb3553dcb279 (DEFAULT_TABLE table=test-table [id=eff81f644e8049529a24bd7003b82e75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:18.160032 17302 tablet_service.cc:1467] Processing CreateTablet for tablet 705d224a4cf64e0abd32fb3553dcb279 (DEFAULT_TABLE table=test-table [id=eff81f644e8049529a24bd7003b82e75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:18.161116 17441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 705d224a4cf64e0abd32fb3553dcb279. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:18.161116 17302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 705d224a4cf64e0abd32fb3553dcb279. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:18.163298 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.135906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58096 (local address 127.15.134.193:38189)
0928 08:30:18.147908 (+ 12002us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:18.147922 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:18.147944 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:30:18.148023 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:18.148028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:18.148216 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:18.148427 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:30:18.149677 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.151049 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.153592 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.155192 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.155362 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.155529 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:18.156513 (+ 984us) server_negotiation.cc:299] Negotiation successful
0928 08:30:18.156707 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11849,"thread_start_us":11729,"threads_started":1}
I20260928 08:30:18.163616 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.135623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:38189 (local address 127.0.0.1:58096)
0928 08:30:18.145227 (+ 9604us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:18.145292 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:18.145506 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:18.148594 (+ 3088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:18.148599 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:18.148664 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:30:18.149431 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.149445 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.151198 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.151206 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.153440 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.153455 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.155691 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.155699 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.155876 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.155913 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 08:30:18.155996 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 08:30:18.162825 (+ 6829us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9232,"mutex_wait_us":49,"thread_start_us":11599,"threads_started":1}
I20260928 08:30:18.167340 17163 tablet_service.cc:1467] Processing CreateTablet for tablet 705d224a4cf64e0abd32fb3553dcb279 (DEFAULT_TABLE table=test-table [id=eff81f644e8049529a24bd7003b82e75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:18.168731 17163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 705d224a4cf64e0abd32fb3553dcb279. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:18.172828 17530 tablet_bootstrap.cc:492] T 705d224a4cf64e0abd32fb3553dcb279 P a2c8ba3451354e5882c303194b0ea691: Bootstrap starting.
I20260928 08:30:18.176054 17530 tablet_bootstrap.cc:654] T 705d224a4cf64e0abd32fb3553dcb279 P a2c8ba3451354e5882c303194b0ea691: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:18.177477 17530 log.cc:824] T 705d224a4cf64e0abd32fb3553dcb279 P a2c8ba3451354e5882c303194b0ea691: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:18.180227 17530 tablet_bootstrap.cc:492] T 705d224a4cf64e0abd32fb3553dcb279 P a2c8ba3451354e5882c303194b0ea691: No bootstrap required, opened a new log
I20260928 08:30:18.180475 17530 ts_tablet_manager.cc:1397] T 705d224a4cf64e0abd32fb3553dcb279 P a2c8ba3451354e5882c303194b0ea691: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260928 08:30:18.182700 17531 tablet_bootstrap.cc:492] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae: Bootstrap starting.
I20260928 08:30:18.183105 17533 tablet_bootstrap.cc:492] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229: Bootstrap starting.
I20260928 08:30:18.186085 17531 tablet_bootstrap.cc:654] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:18.187074 17533 tablet_bootstrap.cc:654] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:18.187196 17531 log.cc:824] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:18.188932 17533 log.cc:824] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:18.192476 17530 raft_consensus.cc:348] T 705d224a4cf64e0abd32fb3553dcb279 P a2c8ba3451354e5882c303194b0ea691 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } }
I20260928 08:30:18.193012 17530 raft_consensus.cc:374] T 705d224a4cf64e0abd32fb3553dcb279 P a2c8ba3451354e5882c303194b0ea691 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:18.193186 17530 raft_consensus.cc:729] T 705d224a4cf64e0abd32fb3553dcb279 P a2c8ba3451354e5882c303194b0ea691 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c8ba3451354e5882c303194b0ea691, State: Initialized, Role: FOLLOWER
I20260928 08:30:18.194008 17530 consensus_queue.cc:260] T 705d224a4cf64e0abd32fb3553dcb279 P a2c8ba3451354e5882c303194b0ea691 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } }
I20260928 08:30:18.195184 17531 tablet_bootstrap.cc:492] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae: No bootstrap required, opened a new log
I20260928 08:30:18.195223 17511 heartbeater.cc:503] Master 127.15.134.254:41709 was elected leader, sending a full tablet report...
I20260928 08:30:18.195520 17531 ts_tablet_manager.cc:1397] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.004s
I20260928 08:30:18.195765 17530 ts_tablet_manager.cc:1428] T 705d224a4cf64e0abd32fb3553dcb279 P a2c8ba3451354e5882c303194b0ea691: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260928 08:30:18.198200 17533 tablet_bootstrap.cc:492] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229: No bootstrap required, opened a new log
I20260928 08:30:18.198506 17533 ts_tablet_manager.cc:1397] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20260928 08:30:18.205724 17531 raft_consensus.cc:348] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } }
I20260928 08:30:18.206153 17531 raft_consensus.cc:374] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:18.206259 17531 raft_consensus.cc:729] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff685508c6c34fa78d511321ed9467ae, State: Initialized, Role: FOLLOWER
I20260928 08:30:18.206812 17531 consensus_queue.cc:260] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } }
I20260928 08:30:18.208149 17372 heartbeater.cc:503] Master 127.15.134.254:41709 was elected leader, sending a full tablet report...
I20260928 08:30:18.208737 17531 ts_tablet_manager.cc:1428] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae: Time spent starting tablet: real 0.013s user 0.006s sys 0.009s
I20260928 08:30:18.208899 17533 raft_consensus.cc:348] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } }
I20260928 08:30:18.209352 17533 raft_consensus.cc:374] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:18.209460 17533 raft_consensus.cc:729] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f40aabf45ef643bfb5ec1cf91fef2229, State: Initialized, Role: FOLLOWER
I20260928 08:30:18.210052 17533 consensus_queue.cc:260] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } }
I20260928 08:30:18.212224 17533 ts_tablet_manager.cc:1428] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260928 08:30:18.218132 17234 tablet.cc:2369] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:30:18.267750 17373 tablet.cc:2369] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:30:18.303925 17512 tablet.cc:2369] T 705d224a4cf64e0abd32fb3553dcb279 P a2c8ba3451354e5882c303194b0ea691: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:30:18.356997 17538 raft_consensus.cc:482] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:30:18.357374 17538 raft_consensus.cc:504] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } }
I20260928 08:30:18.358862 17537 raft_consensus.cc:482] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:30:18.359351 17538 leader_election.cc:288] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2c8ba3451354e5882c303194b0ea691 (127.15.134.195:45483), ff685508c6c34fa78d511321ed9467ae (127.15.134.194:33293)
I20260928 08:30:18.359360 17537 raft_consensus.cc:504] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } }
I20260928 08:30:18.362637 17537 leader_election.cc:288] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f40aabf45ef643bfb5ec1cf91fef2229 (127.15.134.193:38189), a2c8ba3451354e5882c303194b0ea691 (127.15.134.195:45483)
I20260928 08:30:18.372484 17529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.360331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:55131 (local address 127.15.134.195:45483)
0928 08:30:18.360670 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:18.360679 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:18.361405 (+ 726us) server_negotiation.cc:407] Connection header received
0928 08:30:18.361762 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:18.361768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:18.361861 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:18.361947 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:30:18.362987 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.364765 (+ 1778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.370069 (+ 5304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.371582 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.371752 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.371832 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:18.372074 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 08:30:18.372197 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260928 08:30:18.372627 17539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.359713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:45483 (local address 127.15.134.193:55131)
0928 08:30:18.361258 (+ 1545us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:18.361304 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:18.361538 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:18.362216 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:18.362220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:18.362261 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:30:18.362757 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.362773 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.364997 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.365005 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.369829 (+ 4824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.369843 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.371784 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.371788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.371901 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.371925 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 08:30:18.372047 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 08:30:18.372181 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1371,"mutex_wait_us":1,"thread_start_us":2003,"threads_started":1}
I20260928 08:30:18.373123 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.360110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:35249 (local address 127.15.134.194:33293)
0928 08:30:18.360575 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:18.360582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:18.362091 (+ 1509us) server_negotiation.cc:407] Connection header received
0928 08:30:18.362333 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:18.362338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:18.362415 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:18.362484 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:30:18.363769 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.365546 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.370422 (+ 4876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.371997 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.372179 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.372263 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:18.372715 (+ 452us) server_negotiation.cc:299] Negotiation successful
0928 08:30:18.372853 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300}
I20260928 08:30:18.373812 17540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.360043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:33293 (local address 127.15.134.193:35249)
0928 08:30:18.361986 (+ 1943us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:18.362000 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:18.362151 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:18.362714 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:18.362720 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:18.362753 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:30:18.363533 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.363548 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.365776 (+ 2228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.365784 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.370218 (+ 4434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.370233 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.372295 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.372303 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.372490 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.372523 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 08:30:18.373297 (+ 774us) client_negotiation.cc:240] Negotiation successful
0928 08:30:18.373442 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1825,"mutex_wait_us":530,"thread_start_us":826,"threads_started":1}
I20260928 08:30:18.373987 17466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705d224a4cf64e0abd32fb3553dcb279" candidate_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c8ba3451354e5882c303194b0ea691" is_pre_election: true
I20260928 08:30:18.374549 17466 raft_consensus.cc:2454] T 705d224a4cf64e0abd32fb3553dcb279 P a2c8ba3451354e5882c303194b0ea691 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f40aabf45ef643bfb5ec1cf91fef2229 in term 0.
I20260928 08:30:18.374902 17327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705d224a4cf64e0abd32fb3553dcb279" candidate_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff685508c6c34fa78d511321ed9467ae" is_pre_election: true
I20260928 08:30:18.375452 17114 leader_election.cc:302] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2c8ba3451354e5882c303194b0ea691, f40aabf45ef643bfb5ec1cf91fef2229; no voters:
I20260928 08:30:18.375607 17327 raft_consensus.cc:2454] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f40aabf45ef643bfb5ec1cf91fef2229 in term 0.
I20260928 08:30:18.376140 17538 raft_consensus.cc:2781] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:30:18.376313 17538 raft_consensus.cc:482] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:30:18.376415 17538 raft_consensus.cc:3037] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:18.378065 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.362290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:58791 (local address 127.15.134.193:38189)
0928 08:30:18.362490 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:18.362496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:18.367261 (+ 4765us) server_negotiation.cc:407] Connection header received
0928 08:30:18.367531 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:18.367536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:18.367640 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:18.367716 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:30:18.368682 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.370245 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.374690 (+ 4445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.376706 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.376924 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.377000 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:18.377661 (+ 661us) server_negotiation.cc:299] Negotiation successful
0928 08:30:18.377804 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 08:30:18.378362 17542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.362173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:38189 (local address 127.15.134.194:58791)
0928 08:30:18.367140 (+ 4967us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:18.367172 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:18.367380 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:18.367916 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:18.367921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:18.367946 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:30:18.368517 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.368533 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.370388 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.370398 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.374515 (+ 4117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.374529 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.377224 (+ 2695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.377232 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.377433 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.377468 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 08:30:18.377557 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 08:30:18.377691 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4810,"thread_start_us":7165,"threads_started":1}
I20260928 08:30:18.379402 17183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705d224a4cf64e0abd32fb3553dcb279" candidate_uuid: "ff685508c6c34fa78d511321ed9467ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" is_pre_election: true
I20260928 08:30:18.379766 17538 raft_consensus.cc:504] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } }
I20260928 08:30:18.380362 17183 raft_consensus.cc:2379] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff685508c6c34fa78d511321ed9467ae in current term 1: Already voted for candidate f40aabf45ef643bfb5ec1cf91fef2229 in this term.
I20260928 08:30:18.381151 17327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705d224a4cf64e0abd32fb3553dcb279" candidate_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff685508c6c34fa78d511321ed9467ae"
I20260928 08:30:18.381363 17466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705d224a4cf64e0abd32fb3553dcb279" candidate_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c8ba3451354e5882c303194b0ea691"
I20260928 08:30:18.381392 17327 raft_consensus.cc:3037] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:18.381608 17466 raft_consensus.cc:3037] T 705d224a4cf64e0abd32fb3553dcb279 P a2c8ba3451354e5882c303194b0ea691 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:18.382122 17538 leader_election.cc:288] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229 [CANDIDATE]: Term 1 election: Requested vote from peers a2c8ba3451354e5882c303194b0ea691 (127.15.134.195:45483), ff685508c6c34fa78d511321ed9467ae (127.15.134.194:33293)
I20260928 08:30:18.383967 17541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.363156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:45483 (local address 127.15.134.194:56991)
0928 08:30:18.369095 (+ 5939us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:18.369128 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:18.369272 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:18.369844 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:18.369849 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:18.369876 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:30:18.371945 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.371962 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.375920 (+ 3958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.375927 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.378729 (+ 2802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.378756 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.382723 (+ 3967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.382733 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.383006 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.383074 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 08:30:18.383201 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 08:30:18.383401 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5730,"thread_start_us":3563,"threads_started":1}
I20260928 08:30:18.386093 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.364085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:56991 (local address 127.15.134.195:45483)
0928 08:30:18.366465 (+ 2380us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:18.366473 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:18.369532 (+ 3059us) server_negotiation.cc:407] Connection header received
0928 08:30:18.369601 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:18.369606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:18.369693 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:18.369801 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:30:18.373048 (+ 3247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.375614 (+ 2566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.379769 (+ 4155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.381524 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.381805 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.381904 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:18.383311 (+ 1407us) server_negotiation.cc:299] Negotiation successful
0928 08:30:18.383503 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2191,"thread_start_us":2007,"threads_started":1}
I20260928 08:30:18.386030 17327 raft_consensus.cc:2454] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f40aabf45ef643bfb5ec1cf91fef2229 in term 1.
I20260928 08:30:18.386139 17466 raft_consensus.cc:2454] T 705d224a4cf64e0abd32fb3553dcb279 P a2c8ba3451354e5882c303194b0ea691 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f40aabf45ef643bfb5ec1cf91fef2229 in term 1.
I20260928 08:30:18.386757 17466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "705d224a4cf64e0abd32fb3553dcb279" candidate_uuid: "ff685508c6c34fa78d511321ed9467ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c8ba3451354e5882c303194b0ea691" is_pre_election: true
I20260928 08:30:18.386749 17114 leader_election.cc:302] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2c8ba3451354e5882c303194b0ea691, f40aabf45ef643bfb5ec1cf91fef2229; no voters:
I20260928 08:30:18.387132 17466 raft_consensus.cc:2379] T 705d224a4cf64e0abd32fb3553dcb279 P a2c8ba3451354e5882c303194b0ea691 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff685508c6c34fa78d511321ed9467ae in current term 1: Already voted for candidate f40aabf45ef643bfb5ec1cf91fef2229 in this term.
I20260928 08:30:18.387420 17538 raft_consensus.cc:2781] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:18.387943 17254 leader_election.cc:302] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff685508c6c34fa78d511321ed9467ae; no voters: a2c8ba3451354e5882c303194b0ea691, f40aabf45ef643bfb5ec1cf91fef2229
I20260928 08:30:18.388876 17537 raft_consensus.cc:2726] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 08:30:18.389191 17538 raft_consensus.cc:686] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229 [term 1 LEADER]: Becoming Leader. State: Replica: f40aabf45ef643bfb5ec1cf91fef2229, State: Running, Role: LEADER
I20260928 08:30:18.389837 17538 consensus_queue.cc:237] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } }
I20260928 08:30:18.395066 17025 catalog_manager.cc:5495] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229 reported cstate change: term changed from 0 to 1, leader changed from <none> to f40aabf45ef643bfb5ec1cf91fef2229 (127.15.134.193). New cstate: current_term: 1 leader_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } health_report { overall_health: UNKNOWN } } }
I20260928 08:30:18.470045 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.463754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58110 (local address 127.15.134.193:38189)
0928 08:30:18.463959 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:18.463964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:18.464120 (+ 156us) server_negotiation.cc:407] Connection header received
0928 08:30:18.464489 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:18.464493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:18.464572 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:18.464763 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:30:18.465201 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.466263 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.467757 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.468165 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.468269 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.468873 (+ 604us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:30:18.469485 (+ 612us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:30:18.469590 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:18.469729 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 08:30:18.469797 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 08:30:18.479067 17466 raft_consensus.cc:1261] T 705d224a4cf64e0abd32fb3553dcb279 P a2c8ba3451354e5882c303194b0ea691 [term 1 FOLLOWER]: Refusing update from remote peer f40aabf45ef643bfb5ec1cf91fef2229: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:30:18.479192 17327 raft_consensus.cc:1261] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae [term 1 FOLLOWER]: Refusing update from remote peer f40aabf45ef643bfb5ec1cf91fef2229: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:30:18.480109 17544 consensus_queue.cc:1035] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:30:18.480533 17538 consensus_queue.cc:1035] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:30:18.495915 17548 mvcc.cc:205] Tried to move back new op lower bound from 7334232958878998528 to 7334232958529110016. Current Snapshot: MvccSnapshot[applied={T|T < 7334232958878998528}]
I20260928 08:30:18.512823 17026 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:30:18.513132 17026 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:30:18.517457 17026 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9fa1c14909246ba87b26d7ff8f71131: Sending DeleteTablet for 3 replicas of tablet 705d224a4cf64e0abd32fb3553dcb279
I20260928 08:30:18.518766 17163 tablet_service.cc:1514] Processing DeleteTablet for tablet 705d224a4cf64e0abd32fb3553dcb279 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:30:18 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:58096
I20260928 08:30:18.518806 17441 tablet_service.cc:1514] Processing DeleteTablet for tablet 705d224a4cf64e0abd32fb3553dcb279 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:30:18 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:56836
I20260928 08:30:18.521109 17302 tablet_service.cc:1514] Processing DeleteTablet for tablet 705d224a4cf64e0abd32fb3553dcb279 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:30:18 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:33890
I20260928 08:30:18.528914 17026 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 08:30:18.531046 17556 tablet_replica.cc:330] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229: stopping tablet replica
I20260928 08:30:18.531559 17556 raft_consensus.cc:2229] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229 [term 1 LEADER]: Raft consensus shutting down.
W20260928 08:30:18.531627 17026 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:30:18.532490 17556 raft_consensus.cc:2258] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:30:18.533345 17557 tablet_replica.cc:330] T 705d224a4cf64e0abd32fb3553dcb279 P a2c8ba3451354e5882c303194b0ea691: stopping tablet replica
I20260928 08:30:18.533151 17558 tablet_replica.cc:330] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae: stopping tablet replica
I20260928 08:30:18.533804 17557 raft_consensus.cc:2229] T 705d224a4cf64e0abd32fb3553dcb279 P a2c8ba3451354e5882c303194b0ea691 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:30:18.533829 17558 raft_consensus.cc:2229] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:30:18.534686 17558 raft_consensus.cc:2258] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:30:18.537168 17558 ts_tablet_manager.cc:1905] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:30:18.541920 17440 tablet_service.cc:1467] Processing CreateTablet for tablet 366c2186299348ef803c9caab3f8b10d (DEFAULT_TABLE table=test-table [id=f6375b15f0c84d63aa0b00ef5e2913c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:18.542461 17440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 366c2186299348ef803c9caab3f8b10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:18.544253 17163 tablet_service.cc:1467] Processing CreateTablet for tablet 366c2186299348ef803c9caab3f8b10d (DEFAULT_TABLE table=test-table [id=f6375b15f0c84d63aa0b00ef5e2913c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:18.544895 17163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 366c2186299348ef803c9caab3f8b10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:18.545362 17302 tablet_service.cc:1467] Processing CreateTablet for tablet 366c2186299348ef803c9caab3f8b10d (DEFAULT_TABLE table=test-table [id=f6375b15f0c84d63aa0b00ef5e2913c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:18.545902 17302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 366c2186299348ef803c9caab3f8b10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:18.549973 17530 tablet_bootstrap.cc:492] T 366c2186299348ef803c9caab3f8b10d P a2c8ba3451354e5882c303194b0ea691: Bootstrap starting.
I20260928 08:30:18.551450 17557 raft_consensus.cc:2258] T 705d224a4cf64e0abd32fb3553dcb279 P a2c8ba3451354e5882c303194b0ea691 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:30:18.552477 17530 tablet_bootstrap.cc:654] T 366c2186299348ef803c9caab3f8b10d P a2c8ba3451354e5882c303194b0ea691: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:18.553164 17556 ts_tablet_manager.cc:1905] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:30:18.557557 17531 tablet_bootstrap.cc:492] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae: Bootstrap starting.
I20260928 08:30:18.558308 17533 tablet_bootstrap.cc:492] T 366c2186299348ef803c9caab3f8b10d P f40aabf45ef643bfb5ec1cf91fef2229: Bootstrap starting.
I20260928 08:30:18.559000 17557 ts_tablet_manager.cc:1905] T 705d224a4cf64e0abd32fb3553dcb279 P a2c8ba3451354e5882c303194b0ea691: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:30:18.559835 17531 tablet_bootstrap.cc:654] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:18.560600 17533 tablet_bootstrap.cc:654] T 366c2186299348ef803c9caab3f8b10d P f40aabf45ef643bfb5ec1cf91fef2229: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:18.566826 17558 ts_tablet_manager.cc:1918] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:30:18.567071 17558 log.cc:1196] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/705d224a4cf64e0abd32fb3553dcb279
I20260928 08:30:18.567762 17558 ts_tablet_manager.cc:1939] T 705d224a4cf64e0abd32fb3553dcb279 P ff685508c6c34fa78d511321ed9467ae: Deleting consensus metadata
I20260928 08:30:18.569516 17557 ts_tablet_manager.cc:1918] T 705d224a4cf64e0abd32fb3553dcb279 P a2c8ba3451354e5882c303194b0ea691: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:30:18.569690 17557 log.cc:1196] T 705d224a4cf64e0abd32fb3553dcb279 P a2c8ba3451354e5882c303194b0ea691: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/705d224a4cf64e0abd32fb3553dcb279
I20260928 08:30:18.570362 17557 ts_tablet_manager.cc:1939] T 705d224a4cf64e0abd32fb3553dcb279 P a2c8ba3451354e5882c303194b0ea691: Deleting consensus metadata
I20260928 08:30:18.571910 17012 catalog_manager.cc:4841] TS ff685508c6c34fa78d511321ed9467ae (127.15.134.194:33293): tablet 705d224a4cf64e0abd32fb3553dcb279 (table test-table [id=eff81f644e8049529a24bd7003b82e75]) successfully deleted
I20260928 08:30:18.572180 17011 catalog_manager.cc:4841] TS a2c8ba3451354e5882c303194b0ea691 (127.15.134.195:45483): tablet 705d224a4cf64e0abd32fb3553dcb279 (table test-table [id=eff81f644e8049529a24bd7003b82e75]) successfully deleted
I20260928 08:30:18.572436 17530 tablet_bootstrap.cc:492] T 366c2186299348ef803c9caab3f8b10d P a2c8ba3451354e5882c303194b0ea691: No bootstrap required, opened a new log
I20260928 08:30:18.572590 17530 ts_tablet_manager.cc:1397] T 366c2186299348ef803c9caab3f8b10d P a2c8ba3451354e5882c303194b0ea691: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.002s
I20260928 08:30:18.573439 17530 raft_consensus.cc:348] T 366c2186299348ef803c9caab3f8b10d P a2c8ba3451354e5882c303194b0ea691 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } }
I20260928 08:30:18.573685 17530 raft_consensus.cc:374] T 366c2186299348ef803c9caab3f8b10d P a2c8ba3451354e5882c303194b0ea691 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:18.573769 17530 raft_consensus.cc:729] T 366c2186299348ef803c9caab3f8b10d P a2c8ba3451354e5882c303194b0ea691 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c8ba3451354e5882c303194b0ea691, State: Initialized, Role: FOLLOWER
I20260928 08:30:18.573812 17531 tablet_bootstrap.cc:492] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae: No bootstrap required, opened a new log
I20260928 08:30:18.573951 17531 ts_tablet_manager.cc:1397] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.004s
I20260928 08:30:18.574034 17530 consensus_queue.cc:260] T 366c2186299348ef803c9caab3f8b10d P a2c8ba3451354e5882c303194b0ea691 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } }
I20260928 08:30:18.574741 17531 raft_consensus.cc:348] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } }
I20260928 08:30:18.574965 17531 raft_consensus.cc:374] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:18.575057 17531 raft_consensus.cc:729] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff685508c6c34fa78d511321ed9467ae, State: Initialized, Role: FOLLOWER
I20260928 08:30:18.575270 17531 consensus_queue.cc:260] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } }
I20260928 08:30:18.575520 17533 tablet_bootstrap.cc:492] T 366c2186299348ef803c9caab3f8b10d P f40aabf45ef643bfb5ec1cf91fef2229: No bootstrap required, opened a new log
I20260928 08:30:18.575670 17533 ts_tablet_manager.cc:1397] T 366c2186299348ef803c9caab3f8b10d P f40aabf45ef643bfb5ec1cf91fef2229: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20260928 08:30:18.575919 17531 ts_tablet_manager.cc:1428] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 08:30:18.576144 17530 ts_tablet_manager.cc:1428] T 366c2186299348ef803c9caab3f8b10d P a2c8ba3451354e5882c303194b0ea691: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 08:30:18.575682 17556 ts_tablet_manager.cc:1918] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:30:18.576553 17556 log.cc:1196] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-0/wal/wals/705d224a4cf64e0abd32fb3553dcb279
I20260928 08:30:18.576578 17533 raft_consensus.cc:348] T 366c2186299348ef803c9caab3f8b10d P f40aabf45ef643bfb5ec1cf91fef2229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } }
I20260928 08:30:18.576838 17533 raft_consensus.cc:374] T 366c2186299348ef803c9caab3f8b10d P f40aabf45ef643bfb5ec1cf91fef2229 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:18.577083 17533 raft_consensus.cc:729] T 366c2186299348ef803c9caab3f8b10d P f40aabf45ef643bfb5ec1cf91fef2229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f40aabf45ef643bfb5ec1cf91fef2229, State: Initialized, Role: FOLLOWER
I20260928 08:30:18.577296 17556 ts_tablet_manager.cc:1939] T 705d224a4cf64e0abd32fb3553dcb279 P f40aabf45ef643bfb5ec1cf91fef2229: Deleting consensus metadata
I20260928 08:30:18.577589 17533 consensus_queue.cc:260] T 366c2186299348ef803c9caab3f8b10d P f40aabf45ef643bfb5ec1cf91fef2229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } }
I20260928 08:30:18.578565 17537 raft_consensus.cc:482] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:30:18.578725 17537 raft_consensus.cc:504] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } }
I20260928 08:30:18.579348 17537 leader_election.cc:288] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f40aabf45ef643bfb5ec1cf91fef2229 (127.15.134.193:38189), a2c8ba3451354e5882c303194b0ea691 (127.15.134.195:45483)
I20260928 08:30:18.578610 17533 ts_tablet_manager.cc:1428] T 366c2186299348ef803c9caab3f8b10d P f40aabf45ef643bfb5ec1cf91fef2229: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:30:18.580504 17466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366c2186299348ef803c9caab3f8b10d" candidate_uuid: "ff685508c6c34fa78d511321ed9467ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c8ba3451354e5882c303194b0ea691" is_pre_election: true
I20260928 08:30:18.580720 17010 catalog_manager.cc:4841] TS f40aabf45ef643bfb5ec1cf91fef2229 (127.15.134.193:38189): tablet 705d224a4cf64e0abd32fb3553dcb279 (table test-table [id=eff81f644e8049529a24bd7003b82e75]) successfully deleted
I20260928 08:30:18.580798 17466 raft_consensus.cc:2454] T 366c2186299348ef803c9caab3f8b10d P a2c8ba3451354e5882c303194b0ea691 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff685508c6c34fa78d511321ed9467ae in term 0.
I20260928 08:30:18.581199 17254 leader_election.cc:302] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2c8ba3451354e5882c303194b0ea691, ff685508c6c34fa78d511321ed9467ae; no voters:
I20260928 08:30:18.581259 17183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366c2186299348ef803c9caab3f8b10d" candidate_uuid: "ff685508c6c34fa78d511321ed9467ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" is_pre_election: true
I20260928 08:30:18.581485 17183 raft_consensus.cc:2454] T 366c2186299348ef803c9caab3f8b10d P f40aabf45ef643bfb5ec1cf91fef2229 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff685508c6c34fa78d511321ed9467ae in term 0.
I20260928 08:30:18.582086 17537 raft_consensus.cc:2781] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:30:18.582270 17537 raft_consensus.cc:482] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:30:18.582432 17537 raft_consensus.cc:3037] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:18.585029 17537 raft_consensus.cc:504] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } }
I20260928 08:30:18.585605 17537 leader_election.cc:288] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae [CANDIDATE]: Term 1 election: Requested vote from peers f40aabf45ef643bfb5ec1cf91fef2229 (127.15.134.193:38189), a2c8ba3451354e5882c303194b0ea691 (127.15.134.195:45483)
I20260928 08:30:18.586045 17183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366c2186299348ef803c9caab3f8b10d" candidate_uuid: "ff685508c6c34fa78d511321ed9467ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40aabf45ef643bfb5ec1cf91fef2229"
I20260928 08:30:18.586150 17466 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "366c2186299348ef803c9caab3f8b10d" candidate_uuid: "ff685508c6c34fa78d511321ed9467ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c8ba3451354e5882c303194b0ea691"
I20260928 08:30:18.586287 17183 raft_consensus.cc:3037] T 366c2186299348ef803c9caab3f8b10d P f40aabf45ef643bfb5ec1cf91fef2229 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:18.586334 17466 raft_consensus.cc:3037] T 366c2186299348ef803c9caab3f8b10d P a2c8ba3451354e5882c303194b0ea691 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:18.589309 17466 raft_consensus.cc:2454] T 366c2186299348ef803c9caab3f8b10d P a2c8ba3451354e5882c303194b0ea691 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff685508c6c34fa78d511321ed9467ae in term 1.
I20260928 08:30:18.589733 17254 leader_election.cc:302] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2c8ba3451354e5882c303194b0ea691, ff685508c6c34fa78d511321ed9467ae; no voters:
I20260928 08:30:18.589732 17183 raft_consensus.cc:2454] T 366c2186299348ef803c9caab3f8b10d P f40aabf45ef643bfb5ec1cf91fef2229 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff685508c6c34fa78d511321ed9467ae in term 1.
I20260928 08:30:18.590071 17537 raft_consensus.cc:2781] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:18.590528 17537 raft_consensus.cc:686] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae [term 1 LEADER]: Becoming Leader. State: Replica: ff685508c6c34fa78d511321ed9467ae, State: Running, Role: LEADER
I20260928 08:30:18.590965 17537 consensus_queue.cc:237] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } }
I20260928 08:30:18.594509 17026 catalog_manager.cc:5495] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae reported cstate change: term changed from 0 to 1, leader changed from <none> to ff685508c6c34fa78d511321ed9467ae (127.15.134.194). New cstate: current_term: 1 leader_uuid: "ff685508c6c34fa78d511321ed9467ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } health_report { overall_health: UNKNOWN } } }
Sep 28 08:30:18 dist-test-slave-qp2n krb5kdc[16981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584216, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:18.654717 17570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.635572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41709 (local address 127.0.0.1:33150)
0928 08:30:18.636715 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:18.636732 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:18.636904 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:18.637234 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:18.637238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:18.638286 (+ 1048us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:18.638551 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.638560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.639898 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.639906 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.641106 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.641118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.641977 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.641984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.642174 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.642939 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:18.642957 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:18.645880 (+ 2923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:18.650878 (+ 4998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:18.650896 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:18.650906 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:18.651589 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:18.652164 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:18.652169 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:18.652172 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:18.652376 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:18.653343 (+ 967us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:18.653354 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:18.653631 (+ 277us) client_negotiation.cc:769] Sending connection context
0928 08:30:18.653873 (+ 242us) client_negotiation.cc:240] Negotiation successful
0928 08:30:18.654241 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"thread_start_us":752,"threads_started":1}
I20260928 08:30:18.654976 17569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.635532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33150 (local address 127.15.134.254:41709)
0928 08:30:18.636540 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:18.636547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:18.636819 (+ 272us) server_negotiation.cc:407] Connection header received
0928 08:30:18.637042 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:18.637045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:18.637099 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:18.637214 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:18.638752 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.639759 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.641244 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.641779 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.641934 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.646111 (+ 4177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:18.646116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:18.646120 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:18.646151 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:18.650689 (+ 4538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:18.651759 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:18.651763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:18.651766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:18.652036 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:18.652595 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:18.652604 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:18.652642 (+ 38us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:18.653104 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:18.653341 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:18.654007 (+ 666us) server_negotiation.cc:299] Negotiation successful
0928 08:30:18.654513 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":238,"server-negotiator.queue_time_us":769,"thread_start_us":451,"threads_started":1}
I20260928 08:30:18.660962 17026 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:33150:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 08:30:18.662055 17026 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:30:18.670121 17163 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc9ebc9a9854240b4a8858cdc6c1969 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=97b6683039b047e3b9f410532f8d78e8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:30:18.670378 17440 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc9ebc9a9854240b4a8858cdc6c1969 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=97b6683039b047e3b9f410532f8d78e8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:30:18.670882 17440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc9ebc9a9854240b4a8858cdc6c1969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:18.670872 17163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc9ebc9a9854240b4a8858cdc6c1969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:18.677292 17302 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc9ebc9a9854240b4a8858cdc6c1969 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=97b6683039b047e3b9f410532f8d78e8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:30:18.677927 17302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc9ebc9a9854240b4a8858cdc6c1969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:18.678750 17533 tablet_bootstrap.cc:492] T 5bc9ebc9a9854240b4a8858cdc6c1969 P f40aabf45ef643bfb5ec1cf91fef2229: Bootstrap starting.
I20260928 08:30:18.678802 17530 tablet_bootstrap.cc:492] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691: Bootstrap starting.
I20260928 08:30:18.680488 17530 tablet_bootstrap.cc:654] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:18.680491 17533 tablet_bootstrap.cc:654] T 5bc9ebc9a9854240b4a8858cdc6c1969 P f40aabf45ef643bfb5ec1cf91fef2229: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:18.685993 17531 tablet_bootstrap.cc:492] T 5bc9ebc9a9854240b4a8858cdc6c1969 P ff685508c6c34fa78d511321ed9467ae: Bootstrap starting.
I20260928 08:30:18.688661 17531 tablet_bootstrap.cc:654] T 5bc9ebc9a9854240b4a8858cdc6c1969 P ff685508c6c34fa78d511321ed9467ae: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:18.692353 17530 tablet_bootstrap.cc:492] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691: No bootstrap required, opened a new log
I20260928 08:30:18.692353 17531 tablet_bootstrap.cc:492] T 5bc9ebc9a9854240b4a8858cdc6c1969 P ff685508c6c34fa78d511321ed9467ae: No bootstrap required, opened a new log
I20260928 08:30:18.692533 17531 ts_tablet_manager.cc:1397] T 5bc9ebc9a9854240b4a8858cdc6c1969 P ff685508c6c34fa78d511321ed9467ae: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260928 08:30:18.692533 17530 ts_tablet_manager.cc:1397] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.006s
I20260928 08:30:18.693328 17530 raft_consensus.cc:348] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } }
I20260928 08:30:18.693326 17531 raft_consensus.cc:348] T 5bc9ebc9a9854240b4a8858cdc6c1969 P ff685508c6c34fa78d511321ed9467ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } }
I20260928 08:30:18.693581 17531 raft_consensus.cc:374] T 5bc9ebc9a9854240b4a8858cdc6c1969 P ff685508c6c34fa78d511321ed9467ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:18.693581 17530 raft_consensus.cc:374] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:18.693667 17531 raft_consensus.cc:729] T 5bc9ebc9a9854240b4a8858cdc6c1969 P ff685508c6c34fa78d511321ed9467ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff685508c6c34fa78d511321ed9467ae, State: Initialized, Role: FOLLOWER
I20260928 08:30:18.693678 17530 raft_consensus.cc:729] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c8ba3451354e5882c303194b0ea691, State: Initialized, Role: FOLLOWER
I20260928 08:30:18.693925 17530 consensus_queue.cc:260] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } }
I20260928 08:30:18.693924 17531 consensus_queue.cc:260] T 5bc9ebc9a9854240b4a8858cdc6c1969 P ff685508c6c34fa78d511321ed9467ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } }
I20260928 08:30:18.694502 17530 ts_tablet_manager.cc:1428] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:30:18.694378 17537 tablet_replica.cc:439] T 5bc9ebc9a9854240b4a8858cdc6c1969 P ff685508c6c34fa78d511321ed9467ae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } } }
I20260928 08:30:18.694675 17537 tablet_replica.cc:442] T 5bc9ebc9a9854240b4a8858cdc6c1969 P ff685508c6c34fa78d511321ed9467ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:30:18.694864 17536 tablet_replica.cc:439] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } } }
I20260928 08:30:18.695178 17536 tablet_replica.cc:442] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:30:18.696184 17531 ts_tablet_manager.cc:1428] T 5bc9ebc9a9854240b4a8858cdc6c1969 P ff685508c6c34fa78d511321ed9467ae: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 08:30:18.696684 17533 tablet_bootstrap.cc:492] T 5bc9ebc9a9854240b4a8858cdc6c1969 P f40aabf45ef643bfb5ec1cf91fef2229: No bootstrap required, opened a new log
I20260928 08:30:18.696878 17533 ts_tablet_manager.cc:1397] T 5bc9ebc9a9854240b4a8858cdc6c1969 P f40aabf45ef643bfb5ec1cf91fef2229: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.011s
I20260928 08:30:18.698228 17533 raft_consensus.cc:348] T 5bc9ebc9a9854240b4a8858cdc6c1969 P f40aabf45ef643bfb5ec1cf91fef2229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } }
I20260928 08:30:18.698477 17533 raft_consensus.cc:374] T 5bc9ebc9a9854240b4a8858cdc6c1969 P f40aabf45ef643bfb5ec1cf91fef2229 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:18.698557 17533 raft_consensus.cc:729] T 5bc9ebc9a9854240b4a8858cdc6c1969 P f40aabf45ef643bfb5ec1cf91fef2229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f40aabf45ef643bfb5ec1cf91fef2229, State: Initialized, Role: FOLLOWER
I20260928 08:30:18.698823 17533 consensus_queue.cc:260] T 5bc9ebc9a9854240b4a8858cdc6c1969 P f40aabf45ef643bfb5ec1cf91fef2229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } }
I20260928 08:30:18.699292 17538 tablet_replica.cc:439] T 5bc9ebc9a9854240b4a8858cdc6c1969 P f40aabf45ef643bfb5ec1cf91fef2229: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } } }
I20260928 08:30:18.699590 17538 tablet_replica.cc:442] T 5bc9ebc9a9854240b4a8858cdc6c1969 P f40aabf45ef643bfb5ec1cf91fef2229: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:30:18.700133 17533 ts_tablet_manager.cc:1428] T 5bc9ebc9a9854240b4a8858cdc6c1969 P f40aabf45ef643bfb5ec1cf91fef2229: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260928 08:30:18.754106 17536 raft_consensus.cc:482] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:30:18.754379 17536 raft_consensus.cc:504] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } }
I20260928 08:30:18.755936 17536 leader_election.cc:288] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f40aabf45ef643bfb5ec1cf91fef2229 (127.15.134.193:38189), ff685508c6c34fa78d511321ed9467ae (127.15.134.194:33293)
I20260928 08:30:18.764896 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.756826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:39401 (local address 127.15.134.193:38189)
0928 08:30:18.757063 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:18.757070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:18.757954 (+ 884us) server_negotiation.cc:407] Connection header received
0928 08:30:18.758119 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:18.758124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:18.758207 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:18.758299 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:30:18.758925 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.760130 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.762326 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.763585 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.763868 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.763951 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:18.764295 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 08:30:18.764476 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 08:30:18.764969 17574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.756227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:38189 (local address 127.15.134.195:39401)
0928 08:30:18.757829 (+ 1602us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:18.757845 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:18.757987 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:18.758333 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:18.758340 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:18.758370 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:30:18.758737 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.758750 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.760278 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.760284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.762175 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.762185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.763871 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.763878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.764072 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.764116 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 08:30:18.764238 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 08:30:18.764453 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1495,"mutex_wait_us":1,"thread_start_us":897,"threads_started":1}
I20260928 08:30:18.765003 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.756335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:42889 (local address 127.15.134.194:33293)
0928 08:30:18.756681 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:18.756688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:18.757104 (+ 416us) server_negotiation.cc:407] Connection header received
0928 08:30:18.757489 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:18.757496 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:18.757578 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:18.757683 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:30:18.758403 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.759380 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.761867 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.763664 (+ 1797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.763883 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.763970 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:18.764532 (+ 562us) server_negotiation.cc:299] Negotiation successful
0928 08:30:18.764718 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260928 08:30:18.765722 17575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.756205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:33293 (local address 127.15.134.195:42889)
0928 08:30:18.756975 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:18.757001 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:18.757250 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:18.757813 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:18.757821 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:18.757848 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:30:18.758214 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.758223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.759519 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.759525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.761681 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.761697 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.764125 (+ 2428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.764131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.764348 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.764384 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 08:30:18.765208 (+ 824us) client_negotiation.cc:240] Negotiation successful
0928 08:30:18.765354 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":26,"thread_start_us":1393,"threads_started":1}
I20260928 08:30:18.766453 17183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc9ebc9a9854240b4a8858cdc6c1969" candidate_uuid: "a2c8ba3451354e5882c303194b0ea691" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" is_pre_election: true
I20260928 08:30:18.766768 17183 raft_consensus.cc:2454] T 5bc9ebc9a9854240b4a8858cdc6c1969 P f40aabf45ef643bfb5ec1cf91fef2229 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c8ba3451354e5882c303194b0ea691 in term 0.
I20260928 08:30:18.767215 17327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc9ebc9a9854240b4a8858cdc6c1969" candidate_uuid: "a2c8ba3451354e5882c303194b0ea691" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff685508c6c34fa78d511321ed9467ae" is_pre_election: true
I20260928 08:30:18.767270 17392 leader_election.cc:302] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2c8ba3451354e5882c303194b0ea691, f40aabf45ef643bfb5ec1cf91fef2229; no voters:
I20260928 08:30:18.767462 17327 raft_consensus.cc:2454] T 5bc9ebc9a9854240b4a8858cdc6c1969 P ff685508c6c34fa78d511321ed9467ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2c8ba3451354e5882c303194b0ea691 in term 0.
I20260928 08:30:18.767733 17536 raft_consensus.cc:2781] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:30:18.767848 17536 raft_consensus.cc:482] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:30:18.767938 17536 raft_consensus.cc:3037] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:18.770700 17536 raft_consensus.cc:504] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } }
I20260928 08:30:18.771308 17536 leader_election.cc:288] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691 [CANDIDATE]: Term 1 election: Requested vote from peers f40aabf45ef643bfb5ec1cf91fef2229 (127.15.134.193:38189), ff685508c6c34fa78d511321ed9467ae (127.15.134.194:33293)
I20260928 08:30:18.771827 17183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc9ebc9a9854240b4a8858cdc6c1969" candidate_uuid: "a2c8ba3451354e5882c303194b0ea691" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40aabf45ef643bfb5ec1cf91fef2229"
I20260928 08:30:18.771821 17327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bc9ebc9a9854240b4a8858cdc6c1969" candidate_uuid: "a2c8ba3451354e5882c303194b0ea691" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff685508c6c34fa78d511321ed9467ae"
I20260928 08:30:18.772029 17327 raft_consensus.cc:3037] T 5bc9ebc9a9854240b4a8858cdc6c1969 P ff685508c6c34fa78d511321ed9467ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:18.772061 17183 raft_consensus.cc:3037] T 5bc9ebc9a9854240b4a8858cdc6c1969 P f40aabf45ef643bfb5ec1cf91fef2229 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:18.775086 17327 raft_consensus.cc:2454] T 5bc9ebc9a9854240b4a8858cdc6c1969 P ff685508c6c34fa78d511321ed9467ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c8ba3451354e5882c303194b0ea691 in term 1.
I20260928 08:30:18.775133 17183 raft_consensus.cc:2454] T 5bc9ebc9a9854240b4a8858cdc6c1969 P f40aabf45ef643bfb5ec1cf91fef2229 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2c8ba3451354e5882c303194b0ea691 in term 1.
I20260928 08:30:18.775528 17394 leader_election.cc:302] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2c8ba3451354e5882c303194b0ea691, ff685508c6c34fa78d511321ed9467ae; no voters:
I20260928 08:30:18.775965 17536 raft_consensus.cc:2781] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:18.776671 17536 raft_consensus.cc:686] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691 [term 1 LEADER]: Becoming Leader. State: Replica: a2c8ba3451354e5882c303194b0ea691, State: Running, Role: LEADER
I20260928 08:30:18.777401 17536 consensus_queue.cc:237] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } }
I20260928 08:30:18.779706 17576 tablet_replica.cc:439] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691: TxnStatusTablet state changed. Reason: New leader a2c8ba3451354e5882c303194b0ea691. Latest consensus state: current_term: 1 leader_uuid: "a2c8ba3451354e5882c303194b0ea691" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } } }
I20260928 08:30:18.780154 17576 tablet_replica.cc:442] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:30:18.781167 17578 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 08:30:18.781407 17578 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 08:30:18.781728 17025 catalog_manager.cc:5495] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2c8ba3451354e5882c303194b0ea691 (127.15.134.195). New cstate: current_term: 1 leader_uuid: "a2c8ba3451354e5882c303194b0ea691" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } health_report { overall_health: HEALTHY } } }
I20260928 08:30:18.805912 17570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.797773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:45483 (local address 127.0.0.1:56840)
0928 08:30:18.798010 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:18.798031 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:18.798181 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:18.798940 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:18.798947 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:18.798960 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:30:18.799179 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.799192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.800696 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.800703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.802582 (+ 1879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.802600 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.803687 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.803694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.803919 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.804006 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:30:18.805295 (+ 1289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:30:18.805317 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 08:30:18.805401 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 08:30:18.805583 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 08:30:18.806555 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.797915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56840 (local address 127.15.134.195:45483)
0928 08:30:18.798100 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:18.798104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:18.798123 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:30:18.798503 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:18.798509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:18.798686 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:18.798903 (+ 217us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:30:18.799346 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.800510 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.802759 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.803441 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.803609 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.804238 (+ 629us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:30:18.805065 (+ 827us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:30:18.805216 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:18.806166 (+ 950us) server_negotiation.cc:299] Negotiation successful
0928 08:30:18.806263 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 08:30:18.811396 17327 raft_consensus.cc:1261] T 5bc9ebc9a9854240b4a8858cdc6c1969 P ff685508c6c34fa78d511321ed9467ae [term 1 FOLLOWER]: Refusing update from remote peer a2c8ba3451354e5882c303194b0ea691: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:30:18.811658 17183 raft_consensus.cc:1261] T 5bc9ebc9a9854240b4a8858cdc6c1969 P f40aabf45ef643bfb5ec1cf91fef2229 [term 1 FOLLOWER]: Refusing update from remote peer a2c8ba3451354e5882c303194b0ea691: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:30:18.812355 17536 consensus_queue.cc:1035] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:30:18.813673 17536 consensus_queue.cc:1035] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:30:18.823190 17538 tablet_replica.cc:439] T 5bc9ebc9a9854240b4a8858cdc6c1969 P f40aabf45ef643bfb5ec1cf91fef2229: TxnStatusTablet state changed. Reason: New leader a2c8ba3451354e5882c303194b0ea691. Latest consensus state: current_term: 1 leader_uuid: "a2c8ba3451354e5882c303194b0ea691" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } } }
I20260928 08:30:18.823505 17538 tablet_replica.cc:442] T 5bc9ebc9a9854240b4a8858cdc6c1969 P f40aabf45ef643bfb5ec1cf91fef2229: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:30:18.825953 17537 tablet_replica.cc:439] T 5bc9ebc9a9854240b4a8858cdc6c1969 P ff685508c6c34fa78d511321ed9467ae: TxnStatusTablet state changed. Reason: New leader a2c8ba3451354e5882c303194b0ea691. Latest consensus state: current_term: 1 leader_uuid: "a2c8ba3451354e5882c303194b0ea691" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } } }
I20260928 08:30:18.826365 17537 tablet_replica.cc:442] T 5bc9ebc9a9854240b4a8858cdc6c1969 P ff685508c6c34fa78d511321ed9467ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:30:18.828847 17536 tablet_replica.cc:439] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a2c8ba3451354e5882c303194b0ea691" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } } }
I20260928 08:30:18.829241 17536 tablet_replica.cc:442] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:30:18.829793 17536 tablet_replica.cc:439] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a2c8ba3451354e5882c303194b0ea691" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } } }
I20260928 08:30:18.830070 17536 tablet_replica.cc:442] T 5bc9ebc9a9854240b4a8858cdc6c1969 P a2c8ba3451354e5882c303194b0ea691: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:30:18.833616 17547 mvcc.cc:205] Tried to move back new op lower bound from 7334232960242892800 to 7334232960116695040. Current Snapshot: MvccSnapshot[applied={T|T < 7334232960242892800}]
I20260928 08:30:18.833953 17538 tablet_replica.cc:439] T 5bc9ebc9a9854240b4a8858cdc6c1969 P f40aabf45ef643bfb5ec1cf91fef2229: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a2c8ba3451354e5882c303194b0ea691" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } } }
I20260928 08:30:18.834257 17538 tablet_replica.cc:442] T 5bc9ebc9a9854240b4a8858cdc6c1969 P f40aabf45ef643bfb5ec1cf91fef2229: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:30:18.837786 17537 tablet_replica.cc:439] T 5bc9ebc9a9854240b4a8858cdc6c1969 P ff685508c6c34fa78d511321ed9467ae: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a2c8ba3451354e5882c303194b0ea691" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 } } peers { permanent_uuid: "ff685508c6c34fa78d511321ed9467ae" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 33293 } } peers { permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 } } }
I20260928 08:30:18.838093 17537 tablet_replica.cc:442] T 5bc9ebc9a9854240b4a8858cdc6c1969 P ff685508c6c34fa78d511321ed9467ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:30:18.852577 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.844529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33906 (local address 127.15.134.194:33293)
0928 08:30:18.844745 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:18.844753 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:18.844774 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:30:18.844861 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:18.844865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:18.844987 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:18.845161 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:30:18.845579 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.846790 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.848964 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.849651 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.849809 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.850471 (+ 662us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:30:18.851273 (+ 802us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:30:18.851426 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:18.851562 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 08:30:18.852116 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 08:30:18.877218 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.868374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:52203 (local address 127.15.134.195:45483)
0928 08:30:18.868562 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:18.868569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:18.869201 (+ 632us) server_negotiation.cc:407] Connection header received
0928 08:30:18.869420 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:18.869425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:18.869532 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:18.869633 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:30:18.870343 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.871599 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.874555 (+ 2956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.875244 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.875399 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.876224 (+ 825us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:30:18.876384 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:30:18.876485 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:18.876751 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 08:30:18.876842 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:30:18.877454 17585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.868251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:45483 (local address 127.15.134.194:52203)
0928 08:30:18.869072 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:18.869099 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:18.869258 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:18.869687 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:18.869692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:18.869705 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:30:18.870154 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.870167 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.871767 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.871773 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.874349 (+ 2576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.874365 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.875579 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.875587 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.875918 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.876000 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:30:18.876500 (+ 500us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:30:18.876513 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 08:30:18.876701 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 08:30:18.876908 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":509,"threads_started":1}
I20260928 08:30:18.890208 17466 raft_consensus.cc:1261] T 366c2186299348ef803c9caab3f8b10d P a2c8ba3451354e5882c303194b0ea691 [term 1 FOLLOWER]: Refusing update from remote peer ff685508c6c34fa78d511321ed9467ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:30:18.890410 17183 raft_consensus.cc:1261] T 366c2186299348ef803c9caab3f8b10d P f40aabf45ef643bfb5ec1cf91fef2229 [term 1 FOLLOWER]: Refusing update from remote peer ff685508c6c34fa78d511321ed9467ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:30:18.891067 17562 consensus_queue.cc:1035] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c8ba3451354e5882c303194b0ea691" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 45483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:30:18.891952 17562 consensus_queue.cc:1035] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40aabf45ef643bfb5ec1cf91fef2229" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:30:18.905279 17549 mvcc.cc:205] Tried to move back new op lower bound from 7334232960566915072 to 7334232959352434688. Current Snapshot: MvccSnapshot[applied={T|T < 7334232960566915072}]
I20260928 08:30:18.961905 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.949883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:49083 (local address 127.15.134.194:33293)
0928 08:30:18.950081 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:18.950088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:18.953603 (+ 3515us) server_negotiation.cc:407] Connection header received
0928 08:30:18.953836 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:18.953841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:18.953923 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:18.954016 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:30:18.955047 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.956294 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.959508 (+ 3214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.960252 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.960405 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.961092 (+ 687us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:30:18.961227 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:30:18.961320 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:18.961506 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 08:30:18.961592 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:30:18.962191 17589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:18.947883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:33293 (local address 127.15.134.195:49083)
0928 08:30:18.953463 (+ 5580us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:18.953494 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:18.953660 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:18.954182 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:18.954187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:18.954200 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:30:18.954808 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.954824 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.956840 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.956846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.959304 (+ 2458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:18.959322 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:18.960446 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:18.960452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:18.960771 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:18.960854 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:30:18.961345 (+ 491us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:30:18.961361 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 08:30:18.961449 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 08:30:18.961603 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5289,"thread_start_us":5044,"threads_started":1}
I20260928 08:30:19.027208 17024 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:30:19.027498 17024 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:30:19.031378 17024 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9fa1c14909246ba87b26d7ff8f71131: Sending DeleteTablet for 3 replicas of tablet 366c2186299348ef803c9caab3f8b10d
I20260928 08:30:19.032757 17163 tablet_service.cc:1514] Processing DeleteTablet for tablet 366c2186299348ef803c9caab3f8b10d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:30:19 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:58096
I20260928 08:30:19.033136 17556 tablet_replica.cc:330] T 366c2186299348ef803c9caab3f8b10d P f40aabf45ef643bfb5ec1cf91fef2229: stopping tablet replica
I20260928 08:30:19.033113 17440 tablet_service.cc:1514] Processing DeleteTablet for tablet 366c2186299348ef803c9caab3f8b10d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:30:19 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:56836
I20260928 08:30:19.033401 17556 raft_consensus.cc:2229] T 366c2186299348ef803c9caab3f8b10d P f40aabf45ef643bfb5ec1cf91fef2229 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:30:19.033479 17557 tablet_replica.cc:330] T 366c2186299348ef803c9caab3f8b10d P a2c8ba3451354e5882c303194b0ea691: stopping tablet replica
I20260928 08:30:19.033653 17556 raft_consensus.cc:2258] T 366c2186299348ef803c9caab3f8b10d P f40aabf45ef643bfb5ec1cf91fef2229 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:30:19.034545 17557 raft_consensus.cc:2229] T 366c2186299348ef803c9caab3f8b10d P a2c8ba3451354e5882c303194b0ea691 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:30:19.034518 17302 tablet_service.cc:1514] Processing DeleteTablet for tablet 366c2186299348ef803c9caab3f8b10d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:30:19 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:33890
I20260928 08:30:19.034866 17558 tablet_replica.cc:330] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae: stopping tablet replica
I20260928 08:30:19.034855 17557 raft_consensus.cc:2258] T 366c2186299348ef803c9caab3f8b10d P a2c8ba3451354e5882c303194b0ea691 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:30:19.035069 17558 raft_consensus.cc:2229] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:30:19.035751 17556 ts_tablet_manager.cc:1905] T 366c2186299348ef803c9caab3f8b10d P f40aabf45ef643bfb5ec1cf91fef2229: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:30:19.036055 17558 raft_consensus.cc:2258] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:30:19.037678 17558 ts_tablet_manager.cc:1905] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:30:19.041883 17557 ts_tablet_manager.cc:1905] T 366c2186299348ef803c9caab3f8b10d P a2c8ba3451354e5882c303194b0ea691: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:30:19.044173 17556 ts_tablet_manager.cc:1918] T 366c2186299348ef803c9caab3f8b10d P f40aabf45ef643bfb5ec1cf91fef2229: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:30:19.044466 17556 log.cc:1196] T 366c2186299348ef803c9caab3f8b10d P f40aabf45ef643bfb5ec1cf91fef2229: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-0/wal/wals/366c2186299348ef803c9caab3f8b10d
I20260928 08:30:19.045125 17556 ts_tablet_manager.cc:1939] T 366c2186299348ef803c9caab3f8b10d P f40aabf45ef643bfb5ec1cf91fef2229: Deleting consensus metadata
I20260928 08:30:19.046150 17558 ts_tablet_manager.cc:1918] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:30:19.046329 17558 log.cc:1196] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/366c2186299348ef803c9caab3f8b10d
I20260928 08:30:19.046793 17558 ts_tablet_manager.cc:1939] T 366c2186299348ef803c9caab3f8b10d P ff685508c6c34fa78d511321ed9467ae: Deleting consensus metadata
I20260928 08:30:19.048174 17010 catalog_manager.cc:4841] TS f40aabf45ef643bfb5ec1cf91fef2229 (127.15.134.193:38189): tablet 366c2186299348ef803c9caab3f8b10d (table test-table [id=f6375b15f0c84d63aa0b00ef5e2913c3]) successfully deleted
I20260928 08:30:19.048519 17012 catalog_manager.cc:4841] TS ff685508c6c34fa78d511321ed9467ae (127.15.134.194:33293): tablet 366c2186299348ef803c9caab3f8b10d (table test-table [id=f6375b15f0c84d63aa0b00ef5e2913c3]) successfully deleted
I20260928 08:30:19.049503 17557 ts_tablet_manager.cc:1918] T 366c2186299348ef803c9caab3f8b10d P a2c8ba3451354e5882c303194b0ea691: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:30:19.049680 17557 log.cc:1196] T 366c2186299348ef803c9caab3f8b10d P a2c8ba3451354e5882c303194b0ea691: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/366c2186299348ef803c9caab3f8b10d
I20260928 08:30:19.050297 17557 ts_tablet_manager.cc:1939] T 366c2186299348ef803c9caab3f8b10d P a2c8ba3451354e5882c303194b0ea691: Deleting consensus metadata
I20260928 08:30:19.051558 17011 catalog_manager.cc:4841] TS a2c8ba3451354e5882c303194b0ea691 (127.15.134.195:45483): tablet 366c2186299348ef803c9caab3f8b10d (table test-table [id=f6375b15f0c84d63aa0b00ef5e2913c3]) successfully deleted
Sep 28 08:30:19 dist-test-slave-qp2n krb5kdc[16981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584218, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.134.193@KRBTEST.COM
I20260928 08:30:19.114835 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:19.057371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58120 (local address 127.15.134.193:38189)
0928 08:30:19.057580 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:19.057587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:19.068494 (+ 10907us) server_negotiation.cc:407] Connection header received
0928 08:30:19.068754 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:19.068760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:19.068886 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:19.069036 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:19.083231 (+ 14195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:19.084462 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:19.086816 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:19.087612 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:19.087757 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:19.105784 (+ 18027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:19.105804 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:19.105824 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:19.105921 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:19.110292 (+ 4371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:19.111327 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:19.111336 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:19.111340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:19.111531 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:19.112012 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:19.112017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:19.112020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:19.112505 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:19.114031 (+ 1526us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:19.114188 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 08:30:19.114431 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260928 08:30:19.116250 17228 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58120
I20260928 08:30:19.161731 17569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:19.137022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33158 (local address 127.15.134.254:41709)
0928 08:30:19.137277 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:19.137284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:19.144143 (+ 6859us) server_negotiation.cc:407] Connection header received
0928 08:30:19.144666 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:19.144673 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:19.144765 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:19.144840 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:19.146786 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:19.147894 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:19.149774 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:19.150480 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:19.150583 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:19.154019 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:19.154026 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:19.154033 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:19.154082 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:19.157954 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:19.159042 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:19.159049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:19.159053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:19.159274 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:19.159761 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:19.159766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:19.159770 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:19.160154 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:19.160401 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:19.161087 (+ 686us) server_negotiation.cc:299] Negotiation successful
0928 08:30:19.161354 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260928 08:30:19.163587 17024 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33158
I20260928 08:30:19.167140 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 17097
I20260928 08:30:19.195942 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:19.169530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58124 (local address 127.15.134.193:38189)
0928 08:30:19.170256 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:19.170262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:19.178719 (+ 8457us) server_negotiation.cc:407] Connection header received
0928 08:30:19.179212 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:19.179218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:19.179301 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:19.179370 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:19.181138 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:19.182246 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:19.184223 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:19.185085 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:19.185193 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:19.188784 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:19.188793 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:19.188797 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:19.188854 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:19.192469 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:19.193402 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:19.193408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:19.193411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:19.193572 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:19.194071 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:19.194076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:19.194080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:19.194454 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:19.194742 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:19.195290 (+ 548us) server_negotiation.cc:299] Negotiation successful
0928 08:30:19.195558 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555}
W20260928 08:30:19.197831 17228 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58124
W20260928 08:30:19.198632 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 17097: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:30:19.198875 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 17097
I20260928 08:30:19.230202 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 17237
Sep 28 08:30:19 dist-test-slave-qp2n krb5kdc[16981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584218, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.134.194@KRBTEST.COM
I20260928 08:30:19.250872 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:19.231012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33920 (local address 127.15.134.194:33293)
0928 08:30:19.231216 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:19.231223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:19.231501 (+ 278us) server_negotiation.cc:407] Connection header received
0928 08:30:19.231686 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:19.231692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:19.231803 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:19.231918 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:19.233539 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:19.234706 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:19.237014 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:19.237681 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:19.237823 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:19.243235 (+ 5412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:19.243251 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:19.243271 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:19.243322 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:19.247444 (+ 4122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:19.248339 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:19.248348 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:19.248352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:19.248512 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:19.249029 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:19.249034 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:19.249038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:19.249519 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:19.249907 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:19.250206 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 08:30:19.250461 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 08:30:19.251648 17367 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33920
W20260928 08:30:19.252403 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 17237: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:30:19.252601 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 17237
I20260928 08:30:19.273056 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 17376
Sep 28 08:30:19 dist-test-slave-qp2n krb5kdc[16981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584218, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.134.195@KRBTEST.COM
I20260928 08:30:19.290326 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:19.273921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56844 (local address 127.15.134.195:45483)
0928 08:30:19.274091 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:19.274097 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:19.274172 (+ 75us) server_negotiation.cc:407] Connection header received
0928 08:30:19.274393 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:19.274398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:19.274467 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:19.274583 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:19.275847 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:19.276846 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:19.278097 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:19.278594 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:19.278747 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:19.283539 (+ 4792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:19.283551 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:19.283568 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:19.283612 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:19.287250 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:19.288067 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:19.288076 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:19.288078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:19.288259 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:19.288703 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:19.288708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:19.288713 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:19.289092 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:19.289455 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:19.289762 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 08:30:19.289988 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 08:30:19.291175 17506 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56844
W20260928 08:30:19.291968 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 17376: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:30:19.292136 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 17376
I20260928 08:30:19.313352 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 16994
I20260928 08:30:19.314714 17066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:19.434697 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 16994
2026-09-28T08:30:19Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4862 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:30:19 dist-test-slave-qp2n krb5kdc[17602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:30:19 dist-test-slave-qp2n krb5kdc[17602](info): set up 2 sockets
Sep 28 08:30:19 dist-test-slave-qp2n krb5kdc[17602](info): commencing operation
krb5kdc: starting...
W20260928 08:30:20.538599 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:30:20 dist-test-slave-qp2n krb5kdc[17602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:30:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:30:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:30:20.734148 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:36493
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:36493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:21.071364 17615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:21.071763 17615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:21.071913 17615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:21.079269 17615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:30:21.079381 17615 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 08:30:21.079481 17615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:21.079597 17615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:21.079674 17615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:30:21.079788 17615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:30:21.090096 17615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:36493
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:36493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:21.091527 17615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:21.093137 17615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:21.106464 17621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:21.106505 17623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:21.106505 17620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:21.107329 17615 server_base.cc:1034] running on GCE node
I20260928 08:30:21.108078 17615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:21.109575 17615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:21.110836 17615 hybrid_clock.cc:648] HybridClock initialized: now 1790584221110768 us; error 60 us; skew 500 ppm
I20260928 08:30:21.111192 17615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:21 dist-test-slave-qp2n krb5kdc[17602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584221, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:21.114413 17615 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:30:21.116043 17615 webserver.cc:460] Webserver started at http://127.15.134.254:44507/ using document root <none> and password file <none>
I20260928 08:30:21.116846 17615 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:21.116979 17615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:21.117293 17615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:21.120292 17615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "42c341dda8ec4da0a76526e04c0d1f8c"
format_stamp: "Formatted at 2026-09-28 08:30:21 on dist-test-slave-qp2n"
server_key: "93ec9f61a81ee7c5519616ee14ca2cf8"
server_key_iv: "e76ea99230d1acc8772c2bf9c78792ec"
server_key_version: "encryptionkey@0"
I20260928 08:30:21.121098 17615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "42c341dda8ec4da0a76526e04c0d1f8c"
format_stamp: "Formatted at 2026-09-28 08:30:21 on dist-test-slave-qp2n"
server_key: "93ec9f61a81ee7c5519616ee14ca2cf8"
server_key_iv: "e76ea99230d1acc8772c2bf9c78792ec"
server_key_version: "encryptionkey@0"
I20260928 08:30:21.126431 17615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:30:21.129894 17630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:21.131170 17615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:30:21.131358 17615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "42c341dda8ec4da0a76526e04c0d1f8c"
format_stamp: "Formatted at 2026-09-28 08:30:21 on dist-test-slave-qp2n"
server_key: "93ec9f61a81ee7c5519616ee14ca2cf8"
server_key_iv: "e76ea99230d1acc8772c2bf9c78792ec"
server_key_version: "encryptionkey@0"
I20260928 08:30:21.131579 17615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:21.153249 17615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:21.157577 17615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:21.157886 17615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:21.179692 17615 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:36493
I20260928 08:30:21.179698 17691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:36493 every 8 connection(s)
I20260928 08:30:21.181151 17615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:30:21.184005 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 17615
I20260928 08:30:21.184216 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:30:21.185084 15899 external_mini_cluster.cc:1428] Setting key b9c6b54b8234cdef7bbc3cc43ee006d2
I20260928 08:30:21.185456 17692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:21.199954 17692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c: Bootstrap starting.
Sep 28 08:30:21 dist-test-slave-qp2n krb5kdc[17602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:21.203280 17692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:21.204413 17692 log.cc:824] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:21.207346 17692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c: No bootstrap required, opened a new log
I20260928 08:30:21.208813 17695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:21.187080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58012 (local address 127.15.134.254:36493)
0928 08:30:21.187840 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:21.187851 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:21.187890 (+ 39us) server_negotiation.cc:407] Connection header received
0928 08:30:21.190067 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:21.190087 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:21.190461 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:21.190790 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:21.192571 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:21.193869 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:21.195600 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:21.196306 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:21.196450 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:21.201265 (+ 4815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:21.201273 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:21.201287 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:21.201339 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:21.204481 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:21.205255 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:21.205270 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:21.205278 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:21.205464 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:21.205866 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:21.205871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:21.205874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:21.206401 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:21.206690 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:21.207180 (+ 490us) server_negotiation.cc:299] Negotiation successful
0928 08:30:21.207750 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":355,"threads_started":1}
I20260928 08:30:21.215129 17692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42c341dda8ec4da0a76526e04c0d1f8c" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 36493 } }
I20260928 08:30:21.215552 17692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:21.215662 17692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42c341dda8ec4da0a76526e04c0d1f8c, State: Initialized, Role: FOLLOWER
I20260928 08:30:21.216238 17692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42c341dda8ec4da0a76526e04c0d1f8c" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 36493 } }
I20260928 08:30:21.216506 17692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:30:21.216706 17692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:30:21.216888 17692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:21.219103 17692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42c341dda8ec4da0a76526e04c0d1f8c" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 36493 } }
I20260928 08:30:21.219656 17692 leader_election.cc:302] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42c341dda8ec4da0a76526e04c0d1f8c; no voters:
I20260928 08:30:21.220329 17692 leader_election.cc:288] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:30:21.220445 17697 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:21.221274 17697 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c [term 1 LEADER]: Becoming Leader. State: Replica: 42c341dda8ec4da0a76526e04c0d1f8c, State: Running, Role: LEADER
I20260928 08:30:21.221873 17697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42c341dda8ec4da0a76526e04c0d1f8c" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 36493 } }
I20260928 08:30:21.222657 17692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:30:21.224671 17698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42c341dda8ec4da0a76526e04c0d1f8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42c341dda8ec4da0a76526e04c0d1f8c" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 36493 } } }
I20260928 08:30:21.225127 17698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:21.225040 17699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42c341dda8ec4da0a76526e04c0d1f8c. Latest consensus state: current_term: 1 leader_uuid: "42c341dda8ec4da0a76526e04c0d1f8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42c341dda8ec4da0a76526e04c0d1f8c" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 36493 } } }
I20260928 08:30:21.225343 17699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:21.228358 17705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:30:21.235461 17705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:30:21.245111 17705 catalog_manager.cc:1347] Generated new cluster ID: 7f506c2d8f7f467cb8d9dc6e3dea6a6f
I20260928 08:30:21.245242 17705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:30:21.268256 17705 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:30:21.270172 17705 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:30:21.279135 17705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c: Generated new TSK 0
I20260928 08:30:21.280197 17705 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.134.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
I20260928 08:30:21.332513 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:0
--local_ip_for_outbound_sockets=127.15.134.193
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:36493
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:21.673271 17718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:21.673623 17718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:21.673817 17718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:21.681419 17718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:21.681589 17718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:21.681766 17718 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 08:30:21.681859 17718 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:30:21.682000 17718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:30:21.691999 17718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.134.254:36493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:21.693471 17718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:21.694864 17718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:21.708437 17723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:21.708747 17724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:21.708760 17726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:21.709811 17718 server_base.cc:1034] running on GCE node
I20260928 08:30:21.710421 17718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:21.711333 17718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:21.712555 17718 hybrid_clock.cc:648] HybridClock initialized: now 1790584221712483 us; error 68 us; skew 500 ppm
I20260928 08:30:21.712896 17718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:21 dist-test-slave-qp2n krb5kdc[17602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584221, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:21.715973 17718 init.cc:377] Logged in from keytab as kudu/127.15.134.193@KRBTEST.COM (short username kudu)
I20260928 08:30:21.717475 17718 webserver.cc:460] Webserver started at http://127.15.134.193:40523/ using document root <none> and password file <none>
I20260928 08:30:21.718173 17718 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:21.718322 17718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:21.718636 17718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:21.721606 17718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-0/data/instance:
uuid: "4d6702619ea54e0e879bb172dc369ba3"
format_stamp: "Formatted at 2026-09-28 08:30:21 on dist-test-slave-qp2n"
server_key: "6379810803119e23ef5e1f75d735d44a"
server_key_iv: "6e6e09d54957756208e8a8adc4d7a347"
server_key_version: "encryptionkey@0"
I20260928 08:30:21.722397 17718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance:
uuid: "4d6702619ea54e0e879bb172dc369ba3"
format_stamp: "Formatted at 2026-09-28 08:30:21 on dist-test-slave-qp2n"
server_key: "6379810803119e23ef5e1f75d735d44a"
server_key_iv: "6e6e09d54957756208e8a8adc4d7a347"
server_key_version: "encryptionkey@0"
I20260928 08:30:21.727483 17718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:30:21.730907 17733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:21.732313 17718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:30:21.732477 17718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "4d6702619ea54e0e879bb172dc369ba3"
format_stamp: "Formatted at 2026-09-28 08:30:21 on dist-test-slave-qp2n"
server_key: "6379810803119e23ef5e1f75d735d44a"
server_key_iv: "6e6e09d54957756208e8a8adc4d7a347"
server_key_version: "encryptionkey@0"
I20260928 08:30:21.732695 17718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:21.746374 17718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:21.750974 17718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:21.751333 17718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:21.754325 17718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:21.754482 17718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:21.754633 17718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:21.754786 17718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 28 08:30:21 dist-test-slave-qp2n krb5kdc[17602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584221, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:21.778312 17742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:21.754954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:42839 (local address 127.15.134.254:36493)
0928 08:30:21.755717 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:21.755723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:21.756877 (+ 1154us) server_negotiation.cc:407] Connection header received
0928 08:30:21.759377 (+ 2500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:21.759384 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:21.759489 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:21.759782 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:21.762377 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:21.763548 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:21.766463 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:21.767069 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:21.767222 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:21.772092 (+ 4870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:21.772098 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:21.772101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:21.772138 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:21.774729 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:21.775528 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:21.775535 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:21.775544 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:21.775687 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:21.776136 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:21.776144 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:21.776148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:21.776427 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:21.776647 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:21.777646 (+ 999us) server_negotiation.cc:299] Negotiation successful
0928 08:30:21.777890 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":501,"threads_started":1}
I20260928 08:30:21.779743 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:21.755695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:36493 (local address 127.15.134.193:42839)
0928 08:30:21.756582 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:21.756669 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:21.759036 (+ 2367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:21.760085 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:21.760097 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:21.761046 (+ 949us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:21.762141 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:21.762173 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:21.763745 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:21.763752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:21.766296 (+ 2544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:21.766318 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:21.767425 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:21.767436 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:21.767836 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:21.768786 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:21.768830 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:21.771831 (+ 3001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:21.774878 (+ 3047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:21.774888 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:21.774906 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:21.775390 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:21.775805 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:21.775810 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:21.775814 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:21.776003 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:21.776604 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:21.776638 (+ 34us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:21.777181 (+ 543us) client_negotiation.cc:769] Sending connection context
0928 08:30:21.777534 (+ 353us) client_negotiation.cc:240] Negotiation successful
0928 08:30:21.778113 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":412,"threads_started":1}
I20260928 08:30:21.806831 17718 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:38513
I20260928 08:30:21.807021 17853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:38513 every 8 connection(s)
I20260928 08:30:21.808472 17718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:30:21.810870 17854 heartbeater.cc:347] Connected to a master server at 127.15.134.254:36493
I20260928 08:30:21.811136 17854 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:21.811825 17854 heartbeater.cc:511] Master 127.15.134.254:36493 requested a full tablet report, sending...
I20260928 08:30:21.813778 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 17718
I20260928 08:30:21.814000 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance
I20260928 08:30:21.814203 17647 ts_manager.cc:194] Registered new tserver with Master: 4d6702619ea54e0e879bb172dc369ba3 (127.15.134.193:38513)
I20260928 08:30:21.814670 17807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:21.788100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:36493 (local address 127.15.134.193:47761)
0928 08:30:21.800494 (+ 12394us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:21.800527 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:21.800813 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:21.801344 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:21.801349 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:21.801833 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:21.802144 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:21.802154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:21.803359 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:21.803363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:21.805088 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:21.805109 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:21.806290 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:21.806301 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:21.806540 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:21.807359 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:21.807378 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:21.808138 (+ 760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:21.811848 (+ 3710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:21.811854 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:21.811858 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:21.812317 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:21.812777 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:21.812781 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:21.812783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:21.812907 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:21.813504 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:21.813509 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:21.813726 (+ 217us) client_negotiation.cc:769] Sending connection context
0928 08:30:21.813891 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 08:30:21.814248 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12137,"thread_start_us":11932,"threads_started":1}
I20260928 08:30:21.814702 15899 external_mini_cluster.cc:1428] Setting key 4953ab22293bb409c574355ffd1ffe60
I20260928 08:30:21.815567 17742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:21.788439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:47761 (local address 127.15.134.254:36493)
0928 08:30:21.788710 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:21.788719 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:21.800700 (+ 11981us) server_negotiation.cc:407] Connection header received
0928 08:30:21.801012 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:21.801020 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:21.801178 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:21.801303 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:21.802314 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:21.803189 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:21.805254 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:21.806032 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:21.806188 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:21.808336 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:21.808342 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:21.808346 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:21.808394 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:21.811711 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:21.812436 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:21.812444 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:21.812449 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:21.812591 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:21.813018 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:21.813024 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:21.813028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:21.813363 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:21.813540 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:21.814964 (+ 1424us) server_negotiation.cc:299] Negotiation successful
0928 08:30:21.815221 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 08:30:21.816502 17647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.193@KRBTEST.COM'} at 127.15.134.193:42839
WARNING: no policy specified for kudu/127.15.134.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
I20260928 08:30:21.861066 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:0
--local_ip_for_outbound_sockets=127.15.134.194
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:36493
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:22.223806 17858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:22.224152 17858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:22.224342 17858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:22.232074 17858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:22.232276 17858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:22.232438 17858 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 08:30:22.232544 17858 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:30:22.232656 17858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:30:22.242897 17858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.134.254:36493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:22.244184 17858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:22.245517 17858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:22.259212 17863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:22.259268 17864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:22.259186 17866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:22.260063 17858 server_base.cc:1034] running on GCE node
I20260928 08:30:22.260794 17858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:22.261871 17858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:22.263105 17858 hybrid_clock.cc:648] HybridClock initialized: now 1790584222263070 us; error 34 us; skew 500 ppm
I20260928 08:30:22.263526 17858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:22 dist-test-slave-qp2n krb5kdc[17602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584222, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:22.266759 17858 init.cc:377] Logged in from keytab as kudu/127.15.134.194@KRBTEST.COM (short username kudu)
I20260928 08:30:22.268254 17858 webserver.cc:460] Webserver started at http://127.15.134.194:36917/ using document root <none> and password file <none>
I20260928 08:30:22.268970 17858 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:22.269076 17858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:22.269344 17858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:22.272135 17858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-1/data/instance:
uuid: "cade9ccb06b54dec9030d39742e83380"
format_stamp: "Formatted at 2026-09-28 08:30:22 on dist-test-slave-qp2n"
server_key: "ef77793939dc601eb8d17774c97d2f98"
server_key_iv: "f3aa3c6137ab1b15227a275a895c5ae8"
server_key_version: "encryptionkey@0"
I20260928 08:30:22.272910 17858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance:
uuid: "cade9ccb06b54dec9030d39742e83380"
format_stamp: "Formatted at 2026-09-28 08:30:22 on dist-test-slave-qp2n"
server_key: "ef77793939dc601eb8d17774c97d2f98"
server_key_iv: "f3aa3c6137ab1b15227a275a895c5ae8"
server_key_version: "encryptionkey@0"
I20260928 08:30:22.278054 17858 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 08:30:22.281682 17873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:22.283214 17858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:30:22.283411 17858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "cade9ccb06b54dec9030d39742e83380"
format_stamp: "Formatted at 2026-09-28 08:30:22 on dist-test-slave-qp2n"
server_key: "ef77793939dc601eb8d17774c97d2f98"
server_key_iv: "f3aa3c6137ab1b15227a275a895c5ae8"
server_key_version: "encryptionkey@0"
I20260928 08:30:22.283633 17858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:22.317178 17858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:22.321800 17858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:22.322117 17858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:22.325124 17858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:22.325281 17858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:22.325426 17858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:22.325560 17858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:30:22 dist-test-slave-qp2n krb5kdc[17602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584222, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:22.360101 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:22.326422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:36493 (local address 127.15.134.194:36947)
0928 08:30:22.327361 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:22.327410 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:22.329549 (+ 2139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:22.330329 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:22.330346 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:22.331166 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:22.331973 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:22.331992 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:22.333967 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:22.333977 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:22.336300 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:22.336314 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:22.337361 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:22.337370 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:22.337674 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:22.338547 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:22.338589 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:22.342166 (+ 3577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:22.346446 (+ 4280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:22.346457 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:22.346476 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:22.354022 (+ 7546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:22.354659 (+ 637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:22.354665 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:22.354669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:22.355053 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:22.355975 (+ 922us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:22.355986 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:22.356744 (+ 758us) client_negotiation.cc:769] Sending connection context
0928 08:30:22.357185 (+ 441us) client_negotiation.cc:240] Negotiation successful
0928 08:30:22.358015 (+ 830us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":420,"threads_started":1}
I20260928 08:30:22.360895 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:22.325705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:36947 (local address 127.15.134.254:36493)
0928 08:30:22.326455 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:22.326461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:22.327553 (+ 1092us) server_negotiation.cc:407] Connection header received
0928 08:30:22.329831 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:22.329838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:22.329928 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:22.330063 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:22.332206 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:22.333764 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:22.336463 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:22.337086 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:22.337246 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:22.342529 (+ 5283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:22.342539 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:22.342544 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:22.342593 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:22.346240 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:22.354233 (+ 7993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:22.354243 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:22.354247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:22.354468 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:22.355250 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:22.355255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:22.355258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:22.355702 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:22.355878 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:22.360172 (+ 4294us) server_negotiation.cc:299] Negotiation successful
0928 08:30:22.360512 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":513,"threads_started":1}
I20260928 08:30:22.392180 17858 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:35915
I20260928 08:30:22.392200 17993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:35915 every 8 connection(s)
I20260928 08:30:22.393766 17858 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:30:22.396504 17994 heartbeater.cc:347] Connected to a master server at 127.15.134.254:36493
I20260928 08:30:22.396813 17994 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:22.397851 17994 heartbeater.cc:511] Master 127.15.134.254:36493 requested a full tablet report, sending...
I20260928 08:30:22.399621 17647 ts_manager.cc:194] Registered new tserver with Master: cade9ccb06b54dec9030d39742e83380 (127.15.134.194:35915)
I20260928 08:30:22.400485 17647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.194@KRBTEST.COM'} at 127.15.134.194:36947
I20260928 08:30:22.402812 17952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:22.371673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:36493 (local address 127.15.134.194:50597)
0928 08:30:22.386611 (+ 14938us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:22.386641 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:22.387321 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:22.387915 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:22.387919 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:22.388450 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:22.388961 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:22.388987 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:22.390341 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:22.390347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:22.391799 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:22.391813 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:22.392925 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:22.392933 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:22.393247 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:22.393973 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:22.393992 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:22.395112 (+ 1120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:22.399560 (+ 4448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:22.399568 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:22.399573 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:22.400049 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:22.400460 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:22.400467 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:22.400471 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:22.400653 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:22.401413 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:22.401420 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:22.401647 (+ 227us) client_negotiation.cc:769] Sending connection context
0928 08:30:22.401798 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 08:30:22.402239 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14736,"thread_start_us":15194,"threads_started":1}
I20260928 08:30:22.402863 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 17858
I20260928 08:30:22.403040 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance
I20260928 08:30:22.403644 15899 external_mini_cluster.cc:1428] Setting key c55d531313f64a3492fb5d5ee35705b2
I20260928 08:30:22.403872 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:22.371700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:50597 (local address 127.15.134.254:36493)
0928 08:30:22.371976 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:22.371986 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:22.386797 (+ 14811us) server_negotiation.cc:407] Connection header received
0928 08:30:22.387547 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:22.387558 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:22.387690 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:22.387876 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:22.389164 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:22.390176 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:22.391964 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:22.392551 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:22.392872 (+ 321us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:22.395379 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:22.395384 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:22.395388 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:22.395423 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:22.399416 (+ 3993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:22.400180 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:22.400189 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:22.400193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:22.400337 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:22.400779 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:22.400786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:22.400789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:22.401254 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:22.403106 (+ 1852us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:22.403269 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 08:30:22.403516 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
WARNING: no policy specified for kudu/127.15.134.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
I20260928 08:30:22.449924 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:0
--local_ip_for_outbound_sockets=127.15.134.195
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:36493
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:45027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:22.792650 17998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:22.792985 17998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:22.793171 17998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:22.800562 17998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:22.800763 17998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:22.800933 17998 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 08:30:22.801021 17998 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:30:22.801147 17998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:30:22.811647 17998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:45027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.134.254:36493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:22.812974 17998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:22.814275 17998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:30:22.819764 17854 heartbeater.cc:503] Master 127.15.134.254:36493 was elected leader, sending a full tablet report...
W20260928 08:30:22.827917 18004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:22.827961 18006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:22.827999 18003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:22.828944 17998 server_base.cc:1034] running on GCE node
I20260928 08:30:22.829604 17998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:22.830619 17998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:22.831947 17998 hybrid_clock.cc:648] HybridClock initialized: now 1790584222831889 us; error 53 us; skew 500 ppm
I20260928 08:30:22.832340 17998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:22 dist-test-slave-qp2n krb5kdc[17602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584222, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:22.835772 17998 init.cc:377] Logged in from keytab as kudu/127.15.134.195@KRBTEST.COM (short username kudu)
I20260928 08:30:22.837530 17998 webserver.cc:460] Webserver started at http://127.15.134.195:32895/ using document root <none> and password file <none>
I20260928 08:30:22.838266 17998 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:22.838402 17998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:22.838743 17998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:22.841866 17998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-2/data/instance:
uuid: "6848fd817df44775ab0088935a3c5f5c"
format_stamp: "Formatted at 2026-09-28 08:30:22 on dist-test-slave-qp2n"
server_key: "c45f9140f4e1317620e0113305bd92a8"
server_key_iv: "3ec9857c15ac82be93792c0af9c638ec"
server_key_version: "encryptionkey@0"
I20260928 08:30:22.842653 17998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance:
uuid: "6848fd817df44775ab0088935a3c5f5c"
format_stamp: "Formatted at 2026-09-28 08:30:22 on dist-test-slave-qp2n"
server_key: "c45f9140f4e1317620e0113305bd92a8"
server_key_iv: "3ec9857c15ac82be93792c0af9c638ec"
server_key_version: "encryptionkey@0"
I20260928 08:30:22.848059 17998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 08:30:22.851981 18013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:22.853449 17998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 08:30:22.853634 17998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "6848fd817df44775ab0088935a3c5f5c"
format_stamp: "Formatted at 2026-09-28 08:30:22 on dist-test-slave-qp2n"
server_key: "c45f9140f4e1317620e0113305bd92a8"
server_key_iv: "3ec9857c15ac82be93792c0af9c638ec"
server_key_version: "encryptionkey@0"
I20260928 08:30:22.853852 17998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:22.881819 17998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:22.886677 17998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:22.887001 17998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:22.890410 17998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:22.890551 17998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:22.890695 17998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:22.890812 17998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:30:22 dist-test-slave-qp2n krb5kdc[17602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584222, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:22.914404 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:22.890778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:49755 (local address 127.15.134.254:36493)
0928 08:30:22.890991 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:22.890999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:22.892626 (+ 1627us) server_negotiation.cc:407] Connection header received
0928 08:30:22.894698 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:22.894705 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:22.894792 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:22.894903 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:22.897230 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:22.898600 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:22.901352 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:22.901984 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:22.902130 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:22.906931 (+ 4801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:22.906939 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:22.906943 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:22.906991 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:22.910151 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:22.911419 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:22.911427 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:22.911431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:22.911614 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:22.912263 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:22.912269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:22.912272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:22.912738 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:22.912890 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:22.913777 (+ 887us) server_negotiation.cc:299] Negotiation successful
0928 08:30:22.913999 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 08:30:22.915719 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:22.891499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:36493 (local address 127.15.134.195:49755)
0928 08:30:22.892403 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:22.892460 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:22.894373 (+ 1913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:22.895191 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:22.895207 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:22.896105 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:22.897002 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:22.897044 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:22.898823 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:22.898828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:22.901199 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:22.901210 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:22.902242 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:22.902250 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:22.902467 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:22.903221 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:22.903253 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:22.906602 (+ 3349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:22.910351 (+ 3749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:22.910368 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:22.910393 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:22.911226 (+ 833us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:22.911749 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:22.911759 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:22.911763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:22.912143 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:22.912856 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:22.912864 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:22.913327 (+ 463us) client_negotiation.cc:769] Sending connection context
0928 08:30:22.913668 (+ 341us) client_negotiation.cc:240] Negotiation successful
0928 08:30:22.914244 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":377,"threads_started":1}
I20260928 08:30:22.940011 18091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:22.923593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:36493 (local address 127.15.134.195:49385)
0928 08:30:22.924534 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:22.924555 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:22.924756 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:22.925188 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:22.925192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:22.925662 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:22.925974 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:22.925983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:22.927720 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:22.927724 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:22.928884 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:22.928899 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:22.929918 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:22.929927 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:22.930169 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:22.931041 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:22.931072 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:22.932413 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:22.936695 (+ 4282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:22.936700 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:22.936708 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:22.937143 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:22.937632 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:22.937637 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:22.937640 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:22.937834 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:22.938656 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:22.938660 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:22.938925 (+ 265us) client_negotiation.cc:769] Sending connection context
0928 08:30:22.939090 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 08:30:22.939472 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"thread_start_us":404,"threads_started":1}
I20260928 08:30:22.941025 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:22.923545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:49385 (local address 127.15.134.254:36493)
0928 08:30:22.923748 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:22.923757 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:22.924646 (+ 889us) server_negotiation.cc:407] Connection header received
0928 08:30:22.924907 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:22.924916 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:22.925031 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:22.925163 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:22.926122 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:22.927557 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:22.929038 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:22.929688 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:22.929833 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:22.932592 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:22.932597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:22.932600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:22.932684 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:22.936514 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:22.937270 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:22.937277 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:22.937283 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:22.937463 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:22.937994 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:22.937998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:22.938000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:22.938445 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:22.938603 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:22.940516 (+ 1913us) server_negotiation.cc:299] Negotiation successful
0928 08:30:22.940750 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:30:22.947055 17998 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:40181
I20260928 08:30:22.947131 18132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:40181 every 8 connection(s)
I20260928 08:30:22.948308 17998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:30:22.950341 18133 heartbeater.cc:347] Connected to a master server at 127.15.134.254:36493
I20260928 08:30:22.950587 18133 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:22.950891 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 17998
I20260928 08:30:22.951064 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance
I20260928 08:30:22.951309 18133 heartbeater.cc:511] Master 127.15.134.254:36493 requested a full tablet report, sending...
I20260928 08:30:22.951601 15899 external_mini_cluster.cc:1428] Setting key ee75bb6adecb1b5c0aca3b192f97b882
I20260928 08:30:22.952786 17647 ts_manager.cc:194] Registered new tserver with Master: 6848fd817df44775ab0088935a3c5f5c (127.15.134.195:40181)
I20260928 08:30:22.953732 17647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.195@KRBTEST.COM'} at 127.15.134.195:49755
I20260928 08:30:22.954830 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 08:30:22 dist-test-slave-qp2n krb5kdc[17602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 08:30:22 dist-test-slave-qp2n krb5kdc[17602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:23.002898 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:22.988725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58022 (local address 127.15.134.254:36493)
0928 08:30:22.988900 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:22.988904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:22.989257 (+ 353us) server_negotiation.cc:407] Connection header received
0928 08:30:22.989407 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:22.989412 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:22.989476 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:22.989569 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:22.991022 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:22.991765 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:22.993033 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:22.993425 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:22.993519 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:22.997582 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:22.997587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:22.997590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:22.997625 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:23.000165 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:23.000907 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:23.000911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:23.000914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:23.001042 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:23.001435 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:23.001440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:23.001442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:23.001716 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:23.001985 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:23.002324 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 08:30:23.002553 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 08:30:23.007179 17645 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:30:23.014137 17645 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:30:23.054884 18150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:23.027287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:38513 (local address 127.0.0.1:59082)
0928 08:30:23.039961 (+ 12674us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:23.040020 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:23.040249 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:23.042812 (+ 2563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:23.042823 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:23.042868 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:30:23.043413 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:23.043429 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.047355 (+ 3926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.047364 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:23.051648 (+ 4284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:23.051662 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.053778 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.053788 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:23.054033 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:23.054098 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 08:30:23.054230 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 08:30:23.054468 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12345,"thread_start_us":23698,"threads_started":1}
I20260928 08:30:23.056174 18149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:23.027822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59082 (local address 127.15.134.193:38513)
0928 08:30:23.041701 (+ 13879us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:23.041715 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:23.041754 (+ 39us) server_negotiation.cc:407] Connection header received
0928 08:30:23.041910 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:23.041921 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:23.042261 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:23.042505 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:30:23.045053 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.047123 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.051827 (+ 4704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.053515 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.055461 (+ 1946us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:23.055642 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:23.055709 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 08:30:23.055886 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13684,"thread_start_us":13549,"threads_started":1}
I20260928 08:30:23.059576 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:23.028748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:35915 (local address 127.0.0.1:58430)
0928 08:30:23.052245 (+ 23497us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:23.052262 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:23.052384 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:23.053099 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:23.053104 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:23.053129 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:30:23.053537 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:23.053549 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.055315 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.055320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:23.056882 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:23.056894 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.058783 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.058789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:23.058959 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:23.058996 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 08:30:23.059102 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 08:30:23.059241 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23359,"thread_start_us":10726,"threads_started":1}
I20260928 08:30:23.059607 18151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:23.029336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58430 (local address 127.15.134.194:35915)
0928 08:30:23.050533 (+ 21197us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:23.050547 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:23.052395 (+ 1848us) server_negotiation.cc:407] Connection header received
0928 08:30:23.052542 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:23.052555 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:23.052892 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:23.053122 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:30:23.053746 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.055165 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.057068 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.058555 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.058738 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:23.058899 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:23.059160 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 08:30:23.059324 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20952,"thread_start_us":21216,"threads_started":1}
I20260928 08:30:23.060563 17784 tablet_service.cc:1467] Processing CreateTablet for tablet 8cee99f0a9bd4d5eace45e522f93fcfd (DEFAULT_TABLE table=test-table [id=0a0796d35fbf44fe8a7f81c43a3f4c1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:23.061502 18146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:23.027317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:40181 (local address 127.0.0.1:57058)
0928 08:30:23.051373 (+ 24056us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:23.051402 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:23.051556 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:23.054120 (+ 2564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:23.054125 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:23.054149 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:30:23.054554 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:23.054567 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.056539 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.056544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:23.058461 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:23.058473 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.060693 (+ 2220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.060699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:23.060861 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:23.060893 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 08:30:23.060984 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 08:30:23.061111 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23867,"thread_start_us":24662,"threads_started":1}
I20260928 08:30:23.062218 17784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cee99f0a9bd4d5eace45e522f93fcfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:23.062263 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:23.027315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57058 (local address 127.15.134.195:40181)
0928 08:30:23.053023 (+ 25708us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:23.053031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:23.053063 (+ 32us) server_negotiation.cc:407] Connection header received
0928 08:30:23.053153 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:23.053159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:23.053640 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:23.053835 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:30:23.054793 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.056359 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.058664 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.060162 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.060359 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:23.060523 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:23.061738 (+ 1215us) server_negotiation.cc:299] Negotiation successful
0928 08:30:23.061913 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25536,"thread_start_us":25348,"threads_started":1}
I20260928 08:30:23.063978 17924 tablet_service.cc:1467] Processing CreateTablet for tablet 8cee99f0a9bd4d5eace45e522f93fcfd (DEFAULT_TABLE table=test-table [id=0a0796d35fbf44fe8a7f81c43a3f4c1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:23.065218 17924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cee99f0a9bd4d5eace45e522f93fcfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:23.065939 18063 tablet_service.cc:1467] Processing CreateTablet for tablet 8cee99f0a9bd4d5eace45e522f93fcfd (DEFAULT_TABLE table=test-table [id=0a0796d35fbf44fe8a7f81c43a3f4c1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:23.068750 18063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cee99f0a9bd4d5eace45e522f93fcfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:23.075793 18152 tablet_bootstrap.cc:492] T 8cee99f0a9bd4d5eace45e522f93fcfd P cade9ccb06b54dec9030d39742e83380: Bootstrap starting.
I20260928 08:30:23.079324 18152 tablet_bootstrap.cc:654] T 8cee99f0a9bd4d5eace45e522f93fcfd P cade9ccb06b54dec9030d39742e83380: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:23.080519 18152 log.cc:824] T 8cee99f0a9bd4d5eace45e522f93fcfd P cade9ccb06b54dec9030d39742e83380: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:23.081983 18153 tablet_bootstrap.cc:492] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3: Bootstrap starting.
I20260928 08:30:23.084117 18152 tablet_bootstrap.cc:492] T 8cee99f0a9bd4d5eace45e522f93fcfd P cade9ccb06b54dec9030d39742e83380: No bootstrap required, opened a new log
I20260928 08:30:23.084414 18152 ts_tablet_manager.cc:1397] T 8cee99f0a9bd4d5eace45e522f93fcfd P cade9ccb06b54dec9030d39742e83380: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260928 08:30:23.085062 18155 tablet_bootstrap.cc:492] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c: Bootstrap starting.
I20260928 08:30:23.085841 18153 tablet_bootstrap.cc:654] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:23.087178 18153 log.cc:824] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:23.088824 18155 tablet_bootstrap.cc:654] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:23.089908 18155 log.cc:824] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:23.091025 18153 tablet_bootstrap.cc:492] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3: No bootstrap required, opened a new log
I20260928 08:30:23.091328 18153 ts_tablet_manager.cc:1397] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260928 08:30:23.092499 18155 tablet_bootstrap.cc:492] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c: No bootstrap required, opened a new log
I20260928 08:30:23.092798 18155 ts_tablet_manager.cc:1397] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260928 08:30:23.095626 18152 raft_consensus.cc:348] T 8cee99f0a9bd4d5eace45e522f93fcfd P cade9ccb06b54dec9030d39742e83380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } } peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } }
I20260928 08:30:23.096078 18152 raft_consensus.cc:374] T 8cee99f0a9bd4d5eace45e522f93fcfd P cade9ccb06b54dec9030d39742e83380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:23.096171 18152 raft_consensus.cc:729] T 8cee99f0a9bd4d5eace45e522f93fcfd P cade9ccb06b54dec9030d39742e83380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cade9ccb06b54dec9030d39742e83380, State: Initialized, Role: FOLLOWER
I20260928 08:30:23.096879 18152 consensus_queue.cc:260] T 8cee99f0a9bd4d5eace45e522f93fcfd P cade9ccb06b54dec9030d39742e83380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } } peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } }
I20260928 08:30:23.097970 17994 heartbeater.cc:503] Master 127.15.134.254:36493 was elected leader, sending a full tablet report...
I20260928 08:30:23.098701 18152 ts_tablet_manager.cc:1428] T 8cee99f0a9bd4d5eace45e522f93fcfd P cade9ccb06b54dec9030d39742e83380: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260928 08:30:23.101816 18153 raft_consensus.cc:348] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } } peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } }
I20260928 08:30:23.101825 18155 raft_consensus.cc:348] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } } peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } }
I20260928 08:30:23.102355 18155 raft_consensus.cc:374] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:23.102355 18153 raft_consensus.cc:374] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:23.102471 18155 raft_consensus.cc:729] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6848fd817df44775ab0088935a3c5f5c, State: Initialized, Role: FOLLOWER
I20260928 08:30:23.102491 18153 raft_consensus.cc:729] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d6702619ea54e0e879bb172dc369ba3, State: Initialized, Role: FOLLOWER
I20260928 08:30:23.103101 18155 consensus_queue.cc:260] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } } peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } }
I20260928 08:30:23.103381 18153 consensus_queue.cc:260] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } } peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } }
I20260928 08:30:23.104779 18133 heartbeater.cc:503] Master 127.15.134.254:36493 was elected leader, sending a full tablet report...
I20260928 08:30:23.105600 18155 ts_tablet_manager.cc:1428] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c: Time spent starting tablet: real 0.013s user 0.010s sys 0.001s
I20260928 08:30:23.107497 18153 ts_tablet_manager.cc:1428] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
W20260928 08:30:23.146036 17995 tablet.cc:2369] T 8cee99f0a9bd4d5eace45e522f93fcfd P cade9ccb06b54dec9030d39742e83380: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:30:23.200104 18134 tablet.cc:2369] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:30:23.201856 18160 raft_consensus.cc:482] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:30:23.202142 18160 raft_consensus.cc:504] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } } peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } }
I20260928 08:30:23.203763 18160 leader_election.cc:288] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6848fd817df44775ab0088935a3c5f5c (127.15.134.195:40181), cade9ccb06b54dec9030d39742e83380 (127.15.134.194:35915)
I20260928 08:30:23.210804 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:23.204136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:57019 (local address 127.15.134.195:40181)
0928 08:30:23.204349 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:23.204356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:23.205433 (+ 1077us) server_negotiation.cc:407] Connection header received
0928 08:30:23.205641 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:23.205649 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:23.205723 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:23.205792 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:30:23.206416 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.207193 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.209121 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.209854 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.210028 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:23.210133 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:23.210411 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 08:30:23.210537 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 08:30:23.210837 18151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:23.204412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:43675 (local address 127.15.134.194:35915)
0928 08:30:23.204673 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:23.204682 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:23.205491 (+ 809us) server_negotiation.cc:407] Connection header received
0928 08:30:23.205691 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:23.205695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:23.205751 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:23.205860 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:30:23.206520 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.207220 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.209095 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.209871 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.210061 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:23.210132 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:23.210452 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 08:30:23.210573 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 08:30:23.210974 18162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:23.204414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:35915 (local address 127.15.134.193:43675)
0928 08:30:23.205346 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:23.205361 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:23.205522 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:23.205893 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:23.205897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:23.205927 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:30:23.206311 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:23.206319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.207377 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.207384 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:23.208933 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:23.208945 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.210125 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.210132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:23.210266 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:23.210294 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 08:30:23.210404 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 08:30:23.210537 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":25,"thread_start_us":500,"threads_started":1}
I20260928 08:30:23.211184 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:23.204342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:40181 (local address 127.15.134.193:57019)
0928 08:30:23.205315 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:23.205331 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:23.205488 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:23.205835 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:23.205843 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:23.205868 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:30:23.206260 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:23.206269 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.207326 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.207332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:23.208938 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:23.208950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.210081 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.210089 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:23.210252 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:23.210279 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 08:30:23.210355 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 08:30:23.210515 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"mutex_wait_us":8,"thread_start_us":566,"threads_started":1}
I20260928 08:30:23.212195 17947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cee99f0a9bd4d5eace45e522f93fcfd" candidate_uuid: "4d6702619ea54e0e879bb172dc369ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cade9ccb06b54dec9030d39742e83380" is_pre_election: true
I20260928 08:30:23.212195 18087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cee99f0a9bd4d5eace45e522f93fcfd" candidate_uuid: "4d6702619ea54e0e879bb172dc369ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6848fd817df44775ab0088935a3c5f5c" is_pre_election: true
I20260928 08:30:23.212555 18159 raft_consensus.cc:482] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:30:23.212812 17947 raft_consensus.cc:2454] T 8cee99f0a9bd4d5eace45e522f93fcfd P cade9ccb06b54dec9030d39742e83380 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d6702619ea54e0e879bb172dc369ba3 in term 0.
I20260928 08:30:23.212845 18159 raft_consensus.cc:504] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } } peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } }
I20260928 08:30:23.213536 17737 leader_election.cc:302] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d6702619ea54e0e879bb172dc369ba3, cade9ccb06b54dec9030d39742e83380; no voters:
I20260928 08:30:23.213606 18087 raft_consensus.cc:2454] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d6702619ea54e0e879bb172dc369ba3 in term 0.
I20260928 08:30:23.213969 18160 raft_consensus.cc:2781] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:30:23.214113 18160 raft_consensus.cc:482] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:30:23.214214 18160 raft_consensus.cc:3037] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:23.214833 18159 leader_election.cc:288] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d6702619ea54e0e879bb172dc369ba3 (127.15.134.193:38513), cade9ccb06b54dec9030d39742e83380 (127.15.134.194:35915)
I20260928 08:30:23.217556 18160 raft_consensus.cc:504] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } } peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } }
I20260928 08:30:23.218302 18160 leader_election.cc:288] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3 [CANDIDATE]: Term 1 election: Requested vote from peers 6848fd817df44775ab0088935a3c5f5c (127.15.134.195:40181), cade9ccb06b54dec9030d39742e83380 (127.15.134.194:35915)
I20260928 08:30:23.218820 18087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cee99f0a9bd4d5eace45e522f93fcfd" candidate_uuid: "4d6702619ea54e0e879bb172dc369ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6848fd817df44775ab0088935a3c5f5c"
I20260928 08:30:23.218907 17947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cee99f0a9bd4d5eace45e522f93fcfd" candidate_uuid: "4d6702619ea54e0e879bb172dc369ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cade9ccb06b54dec9030d39742e83380"
I20260928 08:30:23.219035 18087 raft_consensus.cc:3037] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:23.219108 17947 raft_consensus.cc:3037] T 8cee99f0a9bd4d5eace45e522f93fcfd P cade9ccb06b54dec9030d39742e83380 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:23.222538 18087 raft_consensus.cc:2454] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d6702619ea54e0e879bb172dc369ba3 in term 1.
I20260928 08:30:23.222683 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:23.215384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:38513 (local address 127.15.134.195:39195)
0928 08:30:23.215644 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:23.215659 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:23.215802 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:23.216309 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:23.216315 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:23.216341 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:30:23.216550 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:23.216561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.217948 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.217980 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:23.220150 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:23.220168 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.221849 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.221854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:23.222061 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:23.222106 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 08:30:23.222207 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 08:30:23.222341 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":1}
I20260928 08:30:23.222621 17947 raft_consensus.cc:2454] T 8cee99f0a9bd4d5eace45e522f93fcfd P cade9ccb06b54dec9030d39742e83380 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d6702619ea54e0e879bb172dc369ba3 in term 1.
I20260928 08:30:23.223004 17735 leader_election.cc:302] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d6702619ea54e0e879bb172dc369ba3, 6848fd817df44775ab0088935a3c5f5c; no voters:
I20260928 08:30:23.223300 18149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:23.215829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:39195 (local address 127.15.134.193:38513)
0928 08:30:23.216019 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:23.216029 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:23.216047 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:30:23.216111 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:23.216116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:23.216183 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:23.216282 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:30:23.216698 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.217774 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.220342 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.221556 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.221700 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:23.221770 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:23.222938 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0928 08:30:23.223072 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 08:30:23.223332 18160 raft_consensus.cc:2781] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:23.223582 18151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:23.215288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:57375 (local address 127.15.134.194:35915)
0928 08:30:23.215455 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:23.215460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:23.216186 (+ 726us) server_negotiation.cc:407] Connection header received
0928 08:30:23.216409 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:23.216415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:23.216491 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:23.216555 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:30:23.217188 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.218072 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.220883 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.222203 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.222398 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:23.222476 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:23.223135 (+ 659us) server_negotiation.cc:299] Negotiation successful
0928 08:30:23.223262 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 08:30:23.224211 18163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:23.215456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:35915 (local address 127.15.134.195:57375)
0928 08:30:23.216093 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:23.216105 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:23.216258 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:23.216753 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:23.216756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:23.216774 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:30:23.217018 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:23.217034 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.218346 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.218352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:23.220707 (+ 2355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:23.220723 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.222692 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.222697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:23.222931 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:23.222962 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 08:30:23.223787 (+ 825us) client_negotiation.cc:240] Negotiation successful
0928 08:30:23.223955 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":915,"threads_started":1}
I20260928 08:30:23.224225 17805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cee99f0a9bd4d5eace45e522f93fcfd" candidate_uuid: "6848fd817df44775ab0088935a3c5f5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d6702619ea54e0e879bb172dc369ba3" is_pre_election: true
I20260928 08:30:23.224880 17947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cee99f0a9bd4d5eace45e522f93fcfd" candidate_uuid: "6848fd817df44775ab0088935a3c5f5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cade9ccb06b54dec9030d39742e83380" is_pre_election: true
I20260928 08:30:23.225157 17947 raft_consensus.cc:2379] T 8cee99f0a9bd4d5eace45e522f93fcfd P cade9ccb06b54dec9030d39742e83380 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6848fd817df44775ab0088935a3c5f5c in current term 1: Already voted for candidate 4d6702619ea54e0e879bb172dc369ba3 in this term.
I20260928 08:30:23.225168 18160 raft_consensus.cc:686] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3 [term 1 LEADER]: Becoming Leader. State: Replica: 4d6702619ea54e0e879bb172dc369ba3, State: Running, Role: LEADER
I20260928 08:30:23.225698 18160 consensus_queue.cc:237] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } } peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } }
I20260928 08:30:23.229705 18016 leader_election.cc:302] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6848fd817df44775ab0088935a3c5f5c; no voters: 4d6702619ea54e0e879bb172dc369ba3, cade9ccb06b54dec9030d39742e83380
I20260928 08:30:23.230175 18159 raft_consensus.cc:2726] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 08:30:23.231279 17645 catalog_manager.cc:5495] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d6702619ea54e0e879bb172dc369ba3 (127.15.134.193). New cstate: current_term: 1 leader_uuid: "4d6702619ea54e0e879bb172dc369ba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } health_report { overall_health: UNKNOWN } } }
W20260928 08:30:23.311909 17855 tablet.cc:2369] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 08:30:23 dist-test-slave-qp2n krb5kdc[17602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584221, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:23.337734 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:23.323381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58028 (local address 127.15.134.254:36493)
0928 08:30:23.323604 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:23.323610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:23.324239 (+ 629us) server_negotiation.cc:407] Connection header received
0928 08:30:23.324454 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:23.324458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:23.324519 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:23.324682 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:23.325845 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.326628 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.327840 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.328273 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.328374 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:23.331903 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:23.331908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:23.331911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:23.331940 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:23.334804 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:23.335529 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:23.335534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:23.335536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:23.335711 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:23.336079 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:23.336086 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:23.336089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:23.336463 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:23.336657 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:23.337054 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 08:30:23.337395 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":113,"server-negotiator.queue_time_us":68}
I20260928 08:30:23.337783 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:23.323455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:36493 (local address 127.0.0.1:58028)
0928 08:30:23.324157 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:23.324172 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:23.324301 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:23.324710 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:23.324713 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:23.325425 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:23.325697 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:23.325706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.326764 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.326768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:23.327708 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:23.327718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.328434 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.328438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:23.328584 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:23.329276 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:23.329293 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:23.331718 (+ 2425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:23.334928 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:23.334943 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:23.334951 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:23.335401 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:23.335829 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:23.335834 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:23.335837 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:23.335977 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:23.336581 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:23.336593 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:23.336791 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 08:30:23.336955 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 08:30:23.337247 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":440,"threads_started":1}
I20260928 08:30:23.343245 17645 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:58028:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 08:30:23.344365 17645 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:30:23.351531 17784 tablet_service.cc:1467] Processing CreateTablet for tablet 1e1fe800875e4e06b6c87cf955751e0f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=718d877dd49d48c8b9fef67175eea97e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:30:23.351545 17924 tablet_service.cc:1467] Processing CreateTablet for tablet 1e1fe800875e4e06b6c87cf955751e0f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=718d877dd49d48c8b9fef67175eea97e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:30:23.351745 18063 tablet_service.cc:1467] Processing CreateTablet for tablet 1e1fe800875e4e06b6c87cf955751e0f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=718d877dd49d48c8b9fef67175eea97e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:30:23.352582 17924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e1fe800875e4e06b6c87cf955751e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:23.352596 17784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e1fe800875e4e06b6c87cf955751e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:23.352731 18063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e1fe800875e4e06b6c87cf955751e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:23.358945 18155 tablet_bootstrap.cc:492] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c: Bootstrap starting.
I20260928 08:30:23.359990 18153 tablet_bootstrap.cc:492] T 1e1fe800875e4e06b6c87cf955751e0f P 4d6702619ea54e0e879bb172dc369ba3: Bootstrap starting.
I20260928 08:30:23.360399 18152 tablet_bootstrap.cc:492] T 1e1fe800875e4e06b6c87cf955751e0f P cade9ccb06b54dec9030d39742e83380: Bootstrap starting.
I20260928 08:30:23.360507 18155 tablet_bootstrap.cc:654] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:23.361780 18153 tablet_bootstrap.cc:654] T 1e1fe800875e4e06b6c87cf955751e0f P 4d6702619ea54e0e879bb172dc369ba3: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:23.362210 18152 tablet_bootstrap.cc:654] T 1e1fe800875e4e06b6c87cf955751e0f P cade9ccb06b54dec9030d39742e83380: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:23.364441 18153 tablet_bootstrap.cc:492] T 1e1fe800875e4e06b6c87cf955751e0f P 4d6702619ea54e0e879bb172dc369ba3: No bootstrap required, opened a new log
I20260928 08:30:23.364594 18153 ts_tablet_manager.cc:1397] T 1e1fe800875e4e06b6c87cf955751e0f P 4d6702619ea54e0e879bb172dc369ba3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 08:30:23.364856 18155 tablet_bootstrap.cc:492] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c: No bootstrap required, opened a new log
I20260928 08:30:23.365530 18153 raft_consensus.cc:348] T 1e1fe800875e4e06b6c87cf955751e0f P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:23.365777 18152 tablet_bootstrap.cc:492] T 1e1fe800875e4e06b6c87cf955751e0f P cade9ccb06b54dec9030d39742e83380: No bootstrap required, opened a new log
I20260928 08:30:23.365758 18153 raft_consensus.cc:374] T 1e1fe800875e4e06b6c87cf955751e0f P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:23.365863 18153 raft_consensus.cc:729] T 1e1fe800875e4e06b6c87cf955751e0f P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d6702619ea54e0e879bb172dc369ba3, State: Initialized, Role: FOLLOWER
I20260928 08:30:23.365934 18152 ts_tablet_manager.cc:1397] T 1e1fe800875e4e06b6c87cf955751e0f P cade9ccb06b54dec9030d39742e83380: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260928 08:30:23.366106 18153 consensus_queue.cc:260] T 1e1fe800875e4e06b6c87cf955751e0f P 4d6702619ea54e0e879bb172dc369ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:23.366514 18160 tablet_replica.cc:439] T 1e1fe800875e4e06b6c87cf955751e0f P 4d6702619ea54e0e879bb172dc369ba3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } } }
I20260928 08:30:23.366683 18153 ts_tablet_manager.cc:1428] T 1e1fe800875e4e06b6c87cf955751e0f P 4d6702619ea54e0e879bb172dc369ba3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 08:30:23.366835 18160 tablet_replica.cc:442] T 1e1fe800875e4e06b6c87cf955751e0f P 4d6702619ea54e0e879bb172dc369ba3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:30:23.366668 18152 raft_consensus.cc:348] T 1e1fe800875e4e06b6c87cf955751e0f P cade9ccb06b54dec9030d39742e83380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:23.366910 18152 raft_consensus.cc:374] T 1e1fe800875e4e06b6c87cf955751e0f P cade9ccb06b54dec9030d39742e83380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:23.367002 18152 raft_consensus.cc:729] T 1e1fe800875e4e06b6c87cf955751e0f P cade9ccb06b54dec9030d39742e83380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cade9ccb06b54dec9030d39742e83380, State: Initialized, Role: FOLLOWER
I20260928 08:30:23.367239 18152 consensus_queue.cc:260] T 1e1fe800875e4e06b6c87cf955751e0f P cade9ccb06b54dec9030d39742e83380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:23.367885 18152 ts_tablet_manager.cc:1428] T 1e1fe800875e4e06b6c87cf955751e0f P cade9ccb06b54dec9030d39742e83380: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:30:23.367775 18158 tablet_replica.cc:439] T 1e1fe800875e4e06b6c87cf955751e0f P cade9ccb06b54dec9030d39742e83380: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } } }
I20260928 08:30:23.368148 18155 ts_tablet_manager.cc:1397] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 08:30:23.368175 18158 tablet_replica.cc:442] T 1e1fe800875e4e06b6c87cf955751e0f P cade9ccb06b54dec9030d39742e83380: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:30:23.368973 18155 raft_consensus.cc:348] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:23.369232 18155 raft_consensus.cc:374] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:23.369333 18155 raft_consensus.cc:729] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6848fd817df44775ab0088935a3c5f5c, State: Initialized, Role: FOLLOWER
I20260928 08:30:23.369567 18155 consensus_queue.cc:260] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:23.370008 18159 tablet_replica.cc:439] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } } }
I20260928 08:30:23.370293 18159 tablet_replica.cc:442] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:30:23.370738 18155 ts_tablet_manager.cc:1428] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:30:23.378907 18159 raft_consensus.cc:482] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:30:23.379158 18159 raft_consensus.cc:504] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:23.379725 18159 leader_election.cc:288] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d6702619ea54e0e879bb172dc369ba3 (127.15.134.193:38513), cade9ccb06b54dec9030d39742e83380 (127.15.134.194:35915)
I20260928 08:30:23.380241 17805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1fe800875e4e06b6c87cf955751e0f" candidate_uuid: "6848fd817df44775ab0088935a3c5f5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d6702619ea54e0e879bb172dc369ba3" is_pre_election: true
I20260928 08:30:23.380241 17947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1fe800875e4e06b6c87cf955751e0f" candidate_uuid: "6848fd817df44775ab0088935a3c5f5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cade9ccb06b54dec9030d39742e83380" is_pre_election: true
I20260928 08:30:23.380512 17805 raft_consensus.cc:2454] T 1e1fe800875e4e06b6c87cf955751e0f P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6848fd817df44775ab0088935a3c5f5c in term 0.
I20260928 08:30:23.380493 17947 raft_consensus.cc:2454] T 1e1fe800875e4e06b6c87cf955751e0f P cade9ccb06b54dec9030d39742e83380 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6848fd817df44775ab0088935a3c5f5c in term 0.
I20260928 08:30:23.380970 18017 leader_election.cc:302] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4d6702619ea54e0e879bb172dc369ba3, 6848fd817df44775ab0088935a3c5f5c, cade9ccb06b54dec9030d39742e83380; no voters:
I20260928 08:30:23.381296 18159 raft_consensus.cc:2781] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:30:23.381428 18159 raft_consensus.cc:482] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:30:23.381551 18159 raft_consensus.cc:3037] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:23.384114 18159 raft_consensus.cc:504] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:23.384765 18159 leader_election.cc:288] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c [CANDIDATE]: Term 1 election: Requested vote from peers 4d6702619ea54e0e879bb172dc369ba3 (127.15.134.193:38513), cade9ccb06b54dec9030d39742e83380 (127.15.134.194:35915)
I20260928 08:30:23.385195 17947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1fe800875e4e06b6c87cf955751e0f" candidate_uuid: "6848fd817df44775ab0088935a3c5f5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cade9ccb06b54dec9030d39742e83380"
I20260928 08:30:23.385241 17805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e1fe800875e4e06b6c87cf955751e0f" candidate_uuid: "6848fd817df44775ab0088935a3c5f5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d6702619ea54e0e879bb172dc369ba3"
I20260928 08:30:23.385433 17947 raft_consensus.cc:3037] T 1e1fe800875e4e06b6c87cf955751e0f P cade9ccb06b54dec9030d39742e83380 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:23.385434 17805 raft_consensus.cc:3037] T 1e1fe800875e4e06b6c87cf955751e0f P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:23.388414 17947 raft_consensus.cc:2454] T 1e1fe800875e4e06b6c87cf955751e0f P cade9ccb06b54dec9030d39742e83380 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6848fd817df44775ab0088935a3c5f5c in term 1.
I20260928 08:30:23.388451 17805 raft_consensus.cc:2454] T 1e1fe800875e4e06b6c87cf955751e0f P 4d6702619ea54e0e879bb172dc369ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6848fd817df44775ab0088935a3c5f5c in term 1.
I20260928 08:30:23.388823 18017 leader_election.cc:302] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6848fd817df44775ab0088935a3c5f5c, cade9ccb06b54dec9030d39742e83380; no voters:
I20260928 08:30:23.389115 18159 raft_consensus.cc:2781] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:23.389577 18159 raft_consensus.cc:686] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c [term 1 LEADER]: Becoming Leader. State: Replica: 6848fd817df44775ab0088935a3c5f5c, State: Running, Role: LEADER
I20260928 08:30:23.390101 18159 consensus_queue.cc:237] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:23.394201 18175 tablet_replica.cc:439] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c: TxnStatusTablet state changed. Reason: New leader 6848fd817df44775ab0088935a3c5f5c. Latest consensus state: current_term: 1 leader_uuid: "6848fd817df44775ab0088935a3c5f5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } } }
I20260928 08:30:23.394498 18175 tablet_replica.cc:442] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:30:23.395141 18177 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 08:30:23.394985 17645 catalog_manager.cc:5495] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6848fd817df44775ab0088935a3c5f5c (127.15.134.195). New cstate: current_term: 1 leader_uuid: "6848fd817df44775ab0088935a3c5f5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } health_report { overall_health: HEALTHY } } }
I20260928 08:30:23.395406 18177 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 08:30:23.419421 18171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:23.413594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:40181 (local address 127.0.0.1:57068)
0928 08:30:23.413847 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:23.413867 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:23.414050 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:23.414535 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:23.414539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:23.414553 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:30:23.414749 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:23.414758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.415849 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.415856 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:23.416956 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:23.416968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.417792 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.417799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:23.417969 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:23.418024 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:30:23.418875 (+ 851us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:30:23.418896 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 08:30:23.418960 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 08:30:23.419115 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 08:30:23.419416 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:23.413597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57068 (local address 127.15.134.195:40181)
0928 08:30:23.413783 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:23.413790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:23.413957 (+ 167us) server_negotiation.cc:407] Connection header received
0928 08:30:23.414256 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:23.414262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:23.414358 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:23.414498 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:30:23.414885 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.415720 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.417103 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.417607 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.417770 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:23.418248 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:30:23.418755 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:30:23.418922 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:23.418992 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 08:30:23.419076 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:30:23.428009 17947 raft_consensus.cc:1261] T 1e1fe800875e4e06b6c87cf955751e0f P cade9ccb06b54dec9030d39742e83380 [term 1 FOLLOWER]: Refusing update from remote peer 6848fd817df44775ab0088935a3c5f5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:30:23.428040 17805 raft_consensus.cc:1261] T 1e1fe800875e4e06b6c87cf955751e0f P 4d6702619ea54e0e879bb172dc369ba3 [term 1 FOLLOWER]: Refusing update from remote peer 6848fd817df44775ab0088935a3c5f5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:30:23.428905 18175 consensus_queue.cc:1035] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:30:23.429262 18159 consensus_queue.cc:1035] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:30:23.441668 18160 tablet_replica.cc:439] T 1e1fe800875e4e06b6c87cf955751e0f P 4d6702619ea54e0e879bb172dc369ba3: TxnStatusTablet state changed. Reason: New leader 6848fd817df44775ab0088935a3c5f5c. Latest consensus state: current_term: 1 leader_uuid: "6848fd817df44775ab0088935a3c5f5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } } }
I20260928 08:30:23.442142 18160 tablet_replica.cc:442] T 1e1fe800875e4e06b6c87cf955751e0f P 4d6702619ea54e0e879bb172dc369ba3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:30:23.443424 18158 tablet_replica.cc:439] T 1e1fe800875e4e06b6c87cf955751e0f P cade9ccb06b54dec9030d39742e83380: TxnStatusTablet state changed. Reason: New leader 6848fd817df44775ab0088935a3c5f5c. Latest consensus state: current_term: 1 leader_uuid: "6848fd817df44775ab0088935a3c5f5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } } }
I20260928 08:30:23.443717 18158 tablet_replica.cc:442] T 1e1fe800875e4e06b6c87cf955751e0f P cade9ccb06b54dec9030d39742e83380: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:30:23.448637 18181 mvcc.cc:205] Tried to move back new op lower bound from 7334232979151896576 to 7334232979009736704. Current Snapshot: MvccSnapshot[applied={T|T < 7334232979151896576}]
I20260928 08:30:23.452111 18160 tablet_replica.cc:439] T 1e1fe800875e4e06b6c87cf955751e0f P 4d6702619ea54e0e879bb172dc369ba3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6848fd817df44775ab0088935a3c5f5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } } }
I20260928 08:30:23.452419 18160 tablet_replica.cc:442] T 1e1fe800875e4e06b6c87cf955751e0f P 4d6702619ea54e0e879bb172dc369ba3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:30:23.453783 18159 tablet_replica.cc:439] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6848fd817df44775ab0088935a3c5f5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } } }
I20260928 08:30:23.454092 18159 tablet_replica.cc:442] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:30:23.456889 18175 tablet_replica.cc:439] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6848fd817df44775ab0088935a3c5f5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } } }
I20260928 08:30:23.457374 18175 tablet_replica.cc:442] T 1e1fe800875e4e06b6c87cf955751e0f P 6848fd817df44775ab0088935a3c5f5c: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:30:23.463716 18158 tablet_replica.cc:439] T 1e1fe800875e4e06b6c87cf955751e0f P cade9ccb06b54dec9030d39742e83380: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6848fd817df44775ab0088935a3c5f5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } } }
I20260928 08:30:23.464005 18158 tablet_replica.cc:442] T 1e1fe800875e4e06b6c87cf955751e0f P cade9ccb06b54dec9030d39742e83380: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:30:23.473678 18149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:23.467135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59084 (local address 127.15.134.193:38513)
0928 08:30:23.467322 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:23.467328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:23.467511 (+ 183us) server_negotiation.cc:407] Connection header received
0928 08:30:23.467711 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:23.467715 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:23.467807 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:23.467926 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:30:23.468344 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.469306 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.471296 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.471718 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.471825 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:23.472392 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:30:23.473042 (+ 650us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:30:23.473162 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:23.473310 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 08:30:23.473379 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:30:23.491060 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:23.485201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:48285 (local address 127.15.134.195:40181)
0928 08:30:23.485441 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:23.485449 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:23.485811 (+ 362us) server_negotiation.cc:407] Connection header received
0928 08:30:23.486033 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:23.486039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:23.486123 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:23.486233 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:30:23.486810 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.487674 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.489231 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.489666 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.489822 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:23.490294 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:30:23.490401 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:30:23.490502 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:23.490646 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 08:30:23.490727 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 08:30:23.491153 18191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:23.484957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:40181 (local address 127.15.134.193:48285)
0928 08:30:23.485693 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:23.485726 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:23.485894 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:23.486258 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:23.486263 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:23.486274 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:30:23.486621 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:23.486633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.487784 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.487793 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:23.489083 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:23.489095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.489851 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.489857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:23.489997 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:23.490056 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:30:23.490503 (+ 447us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:30:23.490516 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 08:30:23.490618 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 08:30:23.490753 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":503,"threads_started":1}
I20260928 08:30:23.501485 17947 raft_consensus.cc:1261] T 8cee99f0a9bd4d5eace45e522f93fcfd P cade9ccb06b54dec9030d39742e83380 [term 1 FOLLOWER]: Refusing update from remote peer 4d6702619ea54e0e879bb172dc369ba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:30:23.501716 18087 raft_consensus.cc:1261] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c [term 1 FOLLOWER]: Refusing update from remote peer 4d6702619ea54e0e879bb172dc369ba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:30:23.502163 18160 consensus_queue.cc:1035] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:30:23.502461 18164 consensus_queue.cc:1035] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:30:23.538647 18149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:23.530980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:45223 (local address 127.15.134.193:38513)
0928 08:30:23.531174 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:23.531180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:23.531878 (+ 698us) server_negotiation.cc:407] Connection header received
0928 08:30:23.532146 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:23.532151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:23.532232 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:23.532326 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:30:23.533437 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.534553 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.536519 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.537260 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.537380 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:23.538027 (+ 647us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:30:23.538137 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:30:23.538208 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:23.538350 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 08:30:23.538409 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 08:30:23.538894 18195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:23.531060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:38513 (local address 127.15.134.195:45223)
0928 08:30:23.531781 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:23.531801 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:23.532009 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:23.532339 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:23.532344 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:23.532355 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:30:23.533245 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:23.533260 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.534742 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.534747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:23.536329 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:23.536342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:23.537579 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:23.537585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:23.537780 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:23.537857 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:30:23.538237 (+ 380us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:30:23.538249 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 08:30:23.538337 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 08:30:23.538484 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":351,"threads_started":1}
I20260928 08:30:23.580240 17646 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:30:23.580581 17646 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:30:23.584962 17646 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c: Sending DeleteTablet for 3 replicas of tablet 8cee99f0a9bd4d5eace45e522f93fcfd
I20260928 08:30:23.586210 17784 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cee99f0a9bd4d5eace45e522f93fcfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:30:23 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:59082
I20260928 08:30:23.586293 18063 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cee99f0a9bd4d5eace45e522f93fcfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:30:23 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:57058
I20260928 08:30:23.587802 17924 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cee99f0a9bd4d5eace45e522f93fcfd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:30:23 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:58430
I20260928 08:30:23.595136 18200 tablet_replica.cc:330] T 8cee99f0a9bd4d5eace45e522f93fcfd P cade9ccb06b54dec9030d39742e83380: stopping tablet replica
I20260928 08:30:23.594993 17646 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:30:23.595744 17646 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:30:23.596475 18200 raft_consensus.cc:2229] T 8cee99f0a9bd4d5eace45e522f93fcfd P cade9ccb06b54dec9030d39742e83380 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:30:23.603358 17783 tablet_service.cc:1467] Processing CreateTablet for tablet 56e3752e3eb549259e9a900cb99842f4 (DEFAULT_TABLE table=test-table [id=630cd574a2e44a6ab5c4b8e8958a3c2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:23.603726 18062 tablet_service.cc:1467] Processing CreateTablet for tablet 56e3752e3eb549259e9a900cb99842f4 (DEFAULT_TABLE table=test-table [id=630cd574a2e44a6ab5c4b8e8958a3c2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:23.603889 17783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56e3752e3eb549259e9a900cb99842f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:23.604274 18062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56e3752e3eb549259e9a900cb99842f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:23.605155 18198 tablet_replica.cc:330] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3: stopping tablet replica
I20260928 08:30:23.605630 18200 raft_consensus.cc:2258] T 8cee99f0a9bd4d5eace45e522f93fcfd P cade9ccb06b54dec9030d39742e83380 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:30:23.606330 18198 raft_consensus.cc:2229] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:30:23.608978 17924 tablet_service.cc:1467] Processing CreateTablet for tablet 56e3752e3eb549259e9a900cb99842f4 (DEFAULT_TABLE table=test-table [id=630cd574a2e44a6ab5c4b8e8958a3c2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:23.609421 17924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56e3752e3eb549259e9a900cb99842f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:23.609774 18198 raft_consensus.cc:2258] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:30:23.611546 18155 tablet_bootstrap.cc:492] T 56e3752e3eb549259e9a900cb99842f4 P 6848fd817df44775ab0088935a3c5f5c: Bootstrap starting.
I20260928 08:30:23.612321 18200 ts_tablet_manager.cc:1905] T 8cee99f0a9bd4d5eace45e522f93fcfd P cade9ccb06b54dec9030d39742e83380: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:30:23.613492 18155 tablet_bootstrap.cc:654] T 56e3752e3eb549259e9a900cb99842f4 P 6848fd817df44775ab0088935a3c5f5c: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:23.614161 18198 ts_tablet_manager.cc:1905] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:30:23.619520 18153 tablet_bootstrap.cc:492] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3: Bootstrap starting.
I20260928 08:30:23.621371 18153 tablet_bootstrap.cc:654] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:23.622673 18152 tablet_bootstrap.cc:492] T 56e3752e3eb549259e9a900cb99842f4 P cade9ccb06b54dec9030d39742e83380: Bootstrap starting.
I20260928 08:30:23.624261 18152 tablet_bootstrap.cc:654] T 56e3752e3eb549259e9a900cb99842f4 P cade9ccb06b54dec9030d39742e83380: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:23.630985 18199 tablet_replica.cc:330] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c: stopping tablet replica
I20260928 08:30:23.631518 18199 raft_consensus.cc:2229] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:30:23.631721 18152 tablet_bootstrap.cc:492] T 56e3752e3eb549259e9a900cb99842f4 P cade9ccb06b54dec9030d39742e83380: No bootstrap required, opened a new log
I20260928 08:30:23.631891 18152 ts_tablet_manager.cc:1397] T 56e3752e3eb549259e9a900cb99842f4 P cade9ccb06b54dec9030d39742e83380: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260928 08:30:23.631942 18199 raft_consensus.cc:2258] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:30:23.632804 18153 tablet_bootstrap.cc:492] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3: No bootstrap required, opened a new log
I20260928 08:30:23.632705 18152 raft_consensus.cc:348] T 56e3752e3eb549259e9a900cb99842f4 P cade9ccb06b54dec9030d39742e83380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:23.632954 18152 raft_consensus.cc:374] T 56e3752e3eb549259e9a900cb99842f4 P cade9ccb06b54dec9030d39742e83380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:23.633034 18152 raft_consensus.cc:729] T 56e3752e3eb549259e9a900cb99842f4 P cade9ccb06b54dec9030d39742e83380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cade9ccb06b54dec9030d39742e83380, State: Initialized, Role: FOLLOWER
I20260928 08:30:23.633214 18153 ts_tablet_manager.cc:1397] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.006s
I20260928 08:30:23.633275 18152 consensus_queue.cc:260] T 56e3752e3eb549259e9a900cb99842f4 P cade9ccb06b54dec9030d39742e83380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:23.633996 18152 ts_tablet_manager.cc:1428] T 56e3752e3eb549259e9a900cb99842f4 P cade9ccb06b54dec9030d39742e83380: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 08:30:23.634202 18153 raft_consensus.cc:348] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:23.634609 18153 raft_consensus.cc:374] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:23.634749 18153 raft_consensus.cc:729] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d6702619ea54e0e879bb172dc369ba3, State: Initialized, Role: FOLLOWER
I20260928 08:30:23.635205 18153 consensus_queue.cc:260] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:23.639282 18153 ts_tablet_manager.cc:1428] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260928 08:30:23.641007 18199 ts_tablet_manager.cc:1905] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:30:23.650250 18199 ts_tablet_manager.cc:1918] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:30:23.650432 18199 log.cc:1196] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/8cee99f0a9bd4d5eace45e522f93fcfd
I20260928 08:30:23.651793 18155 tablet_bootstrap.cc:492] T 56e3752e3eb549259e9a900cb99842f4 P 6848fd817df44775ab0088935a3c5f5c: No bootstrap required, opened a new log
I20260928 08:30:23.651980 18155 ts_tablet_manager.cc:1397] T 56e3752e3eb549259e9a900cb99842f4 P 6848fd817df44775ab0088935a3c5f5c: Time spent bootstrapping tablet: real 0.040s user 0.011s sys 0.004s
I20260928 08:30:23.652835 18155 raft_consensus.cc:348] T 56e3752e3eb549259e9a900cb99842f4 P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:23.653091 18155 raft_consensus.cc:374] T 56e3752e3eb549259e9a900cb99842f4 P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:23.653191 18155 raft_consensus.cc:729] T 56e3752e3eb549259e9a900cb99842f4 P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6848fd817df44775ab0088935a3c5f5c, State: Initialized, Role: FOLLOWER
I20260928 08:30:23.653440 18155 consensus_queue.cc:260] T 56e3752e3eb549259e9a900cb99842f4 P 6848fd817df44775ab0088935a3c5f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:23.654043 18155 ts_tablet_manager.cc:1428] T 56e3752e3eb549259e9a900cb99842f4 P 6848fd817df44775ab0088935a3c5f5c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:30:23.655452 18198 ts_tablet_manager.cc:1918] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:30:23.656311 18200 ts_tablet_manager.cc:1918] T 8cee99f0a9bd4d5eace45e522f93fcfd P cade9ccb06b54dec9030d39742e83380: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:30:23.656504 18200 log.cc:1196] T 8cee99f0a9bd4d5eace45e522f93fcfd P cade9ccb06b54dec9030d39742e83380: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/8cee99f0a9bd4d5eace45e522f93fcfd
I20260928 08:30:23.657086 18200 ts_tablet_manager.cc:1939] T 8cee99f0a9bd4d5eace45e522f93fcfd P cade9ccb06b54dec9030d39742e83380: Deleting consensus metadata
I20260928 08:30:23.657299 18198 log.cc:1196] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-0/wal/wals/8cee99f0a9bd4d5eace45e522f93fcfd
I20260928 08:30:23.657913 18198 ts_tablet_manager.cc:1939] T 8cee99f0a9bd4d5eace45e522f93fcfd P 4d6702619ea54e0e879bb172dc369ba3: Deleting consensus metadata
I20260928 08:30:23.658690 18199 ts_tablet_manager.cc:1939] T 8cee99f0a9bd4d5eace45e522f93fcfd P 6848fd817df44775ab0088935a3c5f5c: Deleting consensus metadata
I20260928 08:30:23.660542 17632 catalog_manager.cc:4841] TS 6848fd817df44775ab0088935a3c5f5c (127.15.134.195:40181): tablet 8cee99f0a9bd4d5eace45e522f93fcfd (table test-table [id=0a0796d35fbf44fe8a7f81c43a3f4c1c]) successfully deleted
I20260928 08:30:23.660573 17633 catalog_manager.cc:4841] TS 4d6702619ea54e0e879bb172dc369ba3 (127.15.134.193:38513): tablet 8cee99f0a9bd4d5eace45e522f93fcfd (table test-table [id=0a0796d35fbf44fe8a7f81c43a3f4c1c]) successfully deleted
I20260928 08:30:23.661886 17634 catalog_manager.cc:4841] TS cade9ccb06b54dec9030d39742e83380 (127.15.134.194:35915): tablet 8cee99f0a9bd4d5eace45e522f93fcfd (table test-table [id=0a0796d35fbf44fe8a7f81c43a3f4c1c]) successfully deleted
I20260928 08:30:23.835181 18160 raft_consensus.cc:482] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:30:23.835457 18160 raft_consensus.cc:504] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:23.836381 18160 leader_election.cc:288] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cade9ccb06b54dec9030d39742e83380 (127.15.134.194:35915), 6848fd817df44775ab0088935a3c5f5c (127.15.134.195:40181)
I20260928 08:30:23.837420 17946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e3752e3eb549259e9a900cb99842f4" candidate_uuid: "4d6702619ea54e0e879bb172dc369ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cade9ccb06b54dec9030d39742e83380" is_pre_election: true
I20260928 08:30:23.837718 17946 raft_consensus.cc:2454] T 56e3752e3eb549259e9a900cb99842f4 P cade9ccb06b54dec9030d39742e83380 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d6702619ea54e0e879bb172dc369ba3 in term 0.
I20260928 08:30:23.839314 18087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e3752e3eb549259e9a900cb99842f4" candidate_uuid: "4d6702619ea54e0e879bb172dc369ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6848fd817df44775ab0088935a3c5f5c" is_pre_election: true
I20260928 08:30:23.839593 18087 raft_consensus.cc:2454] T 56e3752e3eb549259e9a900cb99842f4 P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d6702619ea54e0e879bb172dc369ba3 in term 0.
I20260928 08:30:23.840296 17735 leader_election.cc:302] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d6702619ea54e0e879bb172dc369ba3, 6848fd817df44775ab0088935a3c5f5c; no voters:
I20260928 08:30:23.841194 18160 raft_consensus.cc:2781] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:30:23.841316 18160 raft_consensus.cc:482] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:30:23.841409 18160 raft_consensus.cc:3037] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:23.845276 18160 raft_consensus.cc:504] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:23.846632 17946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e3752e3eb549259e9a900cb99842f4" candidate_uuid: "4d6702619ea54e0e879bb172dc369ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cade9ccb06b54dec9030d39742e83380"
I20260928 08:30:23.846848 17946 raft_consensus.cc:3037] T 56e3752e3eb549259e9a900cb99842f4 P cade9ccb06b54dec9030d39742e83380 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:23.848806 18160 leader_election.cc:288] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3 [CANDIDATE]: Term 1 election: Requested vote from peers cade9ccb06b54dec9030d39742e83380 (127.15.134.194:35915), 6848fd817df44775ab0088935a3c5f5c (127.15.134.195:40181)
I20260928 08:30:23.849686 18087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56e3752e3eb549259e9a900cb99842f4" candidate_uuid: "4d6702619ea54e0e879bb172dc369ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6848fd817df44775ab0088935a3c5f5c"
I20260928 08:30:23.849934 18087 raft_consensus.cc:3037] T 56e3752e3eb549259e9a900cb99842f4 P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:23.850069 17946 raft_consensus.cc:2454] T 56e3752e3eb549259e9a900cb99842f4 P cade9ccb06b54dec9030d39742e83380 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d6702619ea54e0e879bb172dc369ba3 in term 1.
I20260928 08:30:23.850605 17737 leader_election.cc:302] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d6702619ea54e0e879bb172dc369ba3, cade9ccb06b54dec9030d39742e83380; no voters:
I20260928 08:30:23.850984 18160 raft_consensus.cc:2781] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:23.851476 18160 raft_consensus.cc:686] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3 [term 1 LEADER]: Becoming Leader. State: Replica: 4d6702619ea54e0e879bb172dc369ba3, State: Running, Role: LEADER
I20260928 08:30:23.851927 18160 consensus_queue.cc:237] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:23.853430 18087 raft_consensus.cc:2454] T 56e3752e3eb549259e9a900cb99842f4 P 6848fd817df44775ab0088935a3c5f5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d6702619ea54e0e879bb172dc369ba3 in term 1.
I20260928 08:30:23.856302 17646 catalog_manager.cc:5495] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d6702619ea54e0e879bb172dc369ba3 (127.15.134.193). New cstate: current_term: 1 leader_uuid: "4d6702619ea54e0e879bb172dc369ba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } health_report { overall_health: UNKNOWN } } }
I20260928 08:30:23.887010 18087 raft_consensus.cc:1261] T 56e3752e3eb549259e9a900cb99842f4 P 6848fd817df44775ab0088935a3c5f5c [term 1 FOLLOWER]: Refusing update from remote peer 4d6702619ea54e0e879bb172dc369ba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:30:23.887691 17947 raft_consensus.cc:1261] T 56e3752e3eb549259e9a900cb99842f4 P cade9ccb06b54dec9030d39742e83380 [term 1 FOLLOWER]: Refusing update from remote peer 4d6702619ea54e0e879bb172dc369ba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:30:23.887810 18164 consensus_queue.cc:1035] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:30:23.888403 18164 consensus_queue.cc:1035] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:30:23.908721 18180 mvcc.cc:205] Tried to move back new op lower bound from 7334232981035593728 to 7334232980900483072. Current Snapshot: MvccSnapshot[applied={T|T < 7334232981035593728}]
W20260928 08:30:23.926402 18181 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 08:30:25.520535 17646 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:30:25.520867 17646 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:30:25.525241 17646 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c: Sending DeleteTablet for 3 replicas of tablet 56e3752e3eb549259e9a900cb99842f4
I20260928 08:30:25.526204 17784 tablet_service.cc:1514] Processing DeleteTablet for tablet 56e3752e3eb549259e9a900cb99842f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:30:25 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:59082
I20260928 08:30:25.526263 17924 tablet_service.cc:1514] Processing DeleteTablet for tablet 56e3752e3eb549259e9a900cb99842f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:30:25 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:58430
I20260928 08:30:25.526444 18062 tablet_service.cc:1514] Processing DeleteTablet for tablet 56e3752e3eb549259e9a900cb99842f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:30:25 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:57058
I20260928 08:30:25.542630 18233 tablet_replica.cc:330] T 56e3752e3eb549259e9a900cb99842f4 P 6848fd817df44775ab0088935a3c5f5c: stopping tablet replica
I20260928 08:30:25.542886 18233 raft_consensus.cc:2229] T 56e3752e3eb549259e9a900cb99842f4 P 6848fd817df44775ab0088935a3c5f5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:30:25.543166 18233 raft_consensus.cc:2258] T 56e3752e3eb549259e9a900cb99842f4 P 6848fd817df44775ab0088935a3c5f5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:30:25.544332 18233 ts_tablet_manager.cc:1905] T 56e3752e3eb549259e9a900cb99842f4 P 6848fd817df44775ab0088935a3c5f5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:30:25.552085 18233 ts_tablet_manager.cc:1918] T 56e3752e3eb549259e9a900cb99842f4 P 6848fd817df44775ab0088935a3c5f5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:30:25.552263 18233 log.cc:1196] T 56e3752e3eb549259e9a900cb99842f4 P 6848fd817df44775ab0088935a3c5f5c: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/56e3752e3eb549259e9a900cb99842f4
I20260928 08:30:25.553074 18231 tablet_replica.cc:330] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3: stopping tablet replica
I20260928 08:30:25.553329 18231 raft_consensus.cc:2229] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:30:25.553434 18233 ts_tablet_manager.cc:1939] T 56e3752e3eb549259e9a900cb99842f4 P 6848fd817df44775ab0088935a3c5f5c: Deleting consensus metadata
I20260928 08:30:25.553388 18232 tablet_replica.cc:330] T 56e3752e3eb549259e9a900cb99842f4 P cade9ccb06b54dec9030d39742e83380: stopping tablet replica
I20260928 08:30:25.553750 18232 raft_consensus.cc:2229] T 56e3752e3eb549259e9a900cb99842f4 P cade9ccb06b54dec9030d39742e83380 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:30:25.553831 18231 raft_consensus.cc:2258] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:30:25.554095 18232 raft_consensus.cc:2258] T 56e3752e3eb549259e9a900cb99842f4 P cade9ccb06b54dec9030d39742e83380 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:30:25.554774 17632 catalog_manager.cc:4841] TS 6848fd817df44775ab0088935a3c5f5c (127.15.134.195:40181): tablet 56e3752e3eb549259e9a900cb99842f4 (table test-table [id=630cd574a2e44a6ab5c4b8e8958a3c2f]) successfully deleted
I20260928 08:30:25.555023 18231 ts_tablet_manager.cc:1905] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:30:25.555258 18232 ts_tablet_manager.cc:1905] T 56e3752e3eb549259e9a900cb99842f4 P cade9ccb06b54dec9030d39742e83380: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:30:25.560752 18232 ts_tablet_manager.cc:1918] T 56e3752e3eb549259e9a900cb99842f4 P cade9ccb06b54dec9030d39742e83380: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:30:25.560835 18231 ts_tablet_manager.cc:1918] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:30:25.560905 18232 log.cc:1196] T 56e3752e3eb549259e9a900cb99842f4 P cade9ccb06b54dec9030d39742e83380: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/56e3752e3eb549259e9a900cb99842f4
I20260928 08:30:25.560997 18231 log.cc:1196] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-0/wal/wals/56e3752e3eb549259e9a900cb99842f4
I20260928 08:30:25.561359 18232 ts_tablet_manager.cc:1939] T 56e3752e3eb549259e9a900cb99842f4 P cade9ccb06b54dec9030d39742e83380: Deleting consensus metadata
I20260928 08:30:25.561400 18231 ts_tablet_manager.cc:1939] T 56e3752e3eb549259e9a900cb99842f4 P 4d6702619ea54e0e879bb172dc369ba3: Deleting consensus metadata
I20260928 08:30:25.562594 17634 catalog_manager.cc:4841] TS cade9ccb06b54dec9030d39742e83380 (127.15.134.194:35915): tablet 56e3752e3eb549259e9a900cb99842f4 (table test-table [id=630cd574a2e44a6ab5c4b8e8958a3c2f]) successfully deleted
I20260928 08:30:25.562594 17633 catalog_manager.cc:4841] TS 4d6702619ea54e0e879bb172dc369ba3 (127.15.134.193:38513): tablet 56e3752e3eb549259e9a900cb99842f4 (table test-table [id=630cd574a2e44a6ab5c4b8e8958a3c2f]) successfully deleted
Sep 28 08:30:25 dist-test-slave-qp2n krb5kdc[17602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584225, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 08:30:25 dist-test-slave-qp2n krb5kdc[17602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584225, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:25.599997 18241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:25.585049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58032 (local address 127.15.134.254:36493)
0928 08:30:25.585710 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:25.585718 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:25.585758 (+ 40us) server_negotiation.cc:407] Connection header received
0928 08:30:25.585980 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:25.585985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:25.586052 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:25.586170 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:25.587479 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:25.588297 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:25.589660 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:25.590059 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:25.590206 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:25.594573 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:25.594579 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:25.594588 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:25.594624 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:25.597355 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:25.598040 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:25.598048 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:25.598053 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:25.598186 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:25.598570 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:25.598576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:25.598580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:25.598867 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:25.599119 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:25.599416 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 08:30:25.599655 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":301,"threads_started":1}
I20260928 08:30:25.603662 17646 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:30:25.604290 17646 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:30:25.611176 17784 tablet_service.cc:1467] Processing CreateTablet for tablet 8093986aee6f4b2bb350eed151051d1b (DEFAULT_TABLE table=test-table [id=de4bfa94f00d492ca5196e6331456e79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:25.611243 17924 tablet_service.cc:1467] Processing CreateTablet for tablet 8093986aee6f4b2bb350eed151051d1b (DEFAULT_TABLE table=test-table [id=de4bfa94f00d492ca5196e6331456e79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:25.611688 17784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8093986aee6f4b2bb350eed151051d1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:25.611718 17924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8093986aee6f4b2bb350eed151051d1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:25.611593 18062 tablet_service.cc:1467] Processing CreateTablet for tablet 8093986aee6f4b2bb350eed151051d1b (DEFAULT_TABLE table=test-table [id=de4bfa94f00d492ca5196e6331456e79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:25.612026 18062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8093986aee6f4b2bb350eed151051d1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:25.617871 18242 tablet_bootstrap.cc:492] T 8093986aee6f4b2bb350eed151051d1b P cade9ccb06b54dec9030d39742e83380: Bootstrap starting.
I20260928 08:30:25.618894 18244 tablet_bootstrap.cc:492] T 8093986aee6f4b2bb350eed151051d1b P 4d6702619ea54e0e879bb172dc369ba3: Bootstrap starting.
I20260928 08:30:25.619885 18242 tablet_bootstrap.cc:654] T 8093986aee6f4b2bb350eed151051d1b P cade9ccb06b54dec9030d39742e83380: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:25.620631 18244 tablet_bootstrap.cc:654] T 8093986aee6f4b2bb350eed151051d1b P 4d6702619ea54e0e879bb172dc369ba3: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:25.623286 18244 tablet_bootstrap.cc:492] T 8093986aee6f4b2bb350eed151051d1b P 4d6702619ea54e0e879bb172dc369ba3: No bootstrap required, opened a new log
I20260928 08:30:25.623539 18244 ts_tablet_manager.cc:1397] T 8093986aee6f4b2bb350eed151051d1b P 4d6702619ea54e0e879bb172dc369ba3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 08:30:25.624594 18244 raft_consensus.cc:348] T 8093986aee6f4b2bb350eed151051d1b P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:25.624934 18244 raft_consensus.cc:374] T 8093986aee6f4b2bb350eed151051d1b P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:25.625034 18244 raft_consensus.cc:729] T 8093986aee6f4b2bb350eed151051d1b P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d6702619ea54e0e879bb172dc369ba3, State: Initialized, Role: FOLLOWER
I20260928 08:30:25.625353 18244 consensus_queue.cc:260] T 8093986aee6f4b2bb350eed151051d1b P 4d6702619ea54e0e879bb172dc369ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:25.626073 18244 ts_tablet_manager.cc:1428] T 8093986aee6f4b2bb350eed151051d1b P 4d6702619ea54e0e879bb172dc369ba3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 08:30:25.627647 18243 tablet_bootstrap.cc:492] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c: Bootstrap starting.
I20260928 08:30:25.628407 18242 tablet_bootstrap.cc:492] T 8093986aee6f4b2bb350eed151051d1b P cade9ccb06b54dec9030d39742e83380: No bootstrap required, opened a new log
I20260928 08:30:25.628574 18242 ts_tablet_manager.cc:1397] T 8093986aee6f4b2bb350eed151051d1b P cade9ccb06b54dec9030d39742e83380: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260928 08:30:25.629388 18243 tablet_bootstrap.cc:654] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:25.629429 18242 raft_consensus.cc:348] T 8093986aee6f4b2bb350eed151051d1b P cade9ccb06b54dec9030d39742e83380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:25.629678 18242 raft_consensus.cc:374] T 8093986aee6f4b2bb350eed151051d1b P cade9ccb06b54dec9030d39742e83380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:25.629783 18242 raft_consensus.cc:729] T 8093986aee6f4b2bb350eed151051d1b P cade9ccb06b54dec9030d39742e83380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cade9ccb06b54dec9030d39742e83380, State: Initialized, Role: FOLLOWER
I20260928 08:30:25.630026 18242 consensus_queue.cc:260] T 8093986aee6f4b2bb350eed151051d1b P cade9ccb06b54dec9030d39742e83380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:25.631794 18243 tablet_bootstrap.cc:492] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c: No bootstrap required, opened a new log
I20260928 08:30:25.632063 18243 ts_tablet_manager.cc:1397] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260928 08:30:25.632445 18242 ts_tablet_manager.cc:1428] T 8093986aee6f4b2bb350eed151051d1b P cade9ccb06b54dec9030d39742e83380: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 08:30:25.632974 18243 raft_consensus.cc:348] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:25.633271 18243 raft_consensus.cc:374] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:25.633373 18243 raft_consensus.cc:729] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6848fd817df44775ab0088935a3c5f5c, State: Initialized, Role: FOLLOWER
I20260928 08:30:25.633637 18243 consensus_queue.cc:260] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:25.634223 18243 ts_tablet_manager.cc:1428] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:30:25.680135 18226 raft_consensus.cc:482] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:30:25.680362 18226 raft_consensus.cc:504] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:25.681182 18226 leader_election.cc:288] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d6702619ea54e0e879bb172dc369ba3 (127.15.134.193:38513), cade9ccb06b54dec9030d39742e83380 (127.15.134.194:35915)
I20260928 08:30:25.681697 17805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8093986aee6f4b2bb350eed151051d1b" candidate_uuid: "6848fd817df44775ab0088935a3c5f5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d6702619ea54e0e879bb172dc369ba3" is_pre_election: true
I20260928 08:30:25.681697 17947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8093986aee6f4b2bb350eed151051d1b" candidate_uuid: "6848fd817df44775ab0088935a3c5f5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cade9ccb06b54dec9030d39742e83380" is_pre_election: true
I20260928 08:30:25.681978 17947 raft_consensus.cc:2454] T 8093986aee6f4b2bb350eed151051d1b P cade9ccb06b54dec9030d39742e83380 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6848fd817df44775ab0088935a3c5f5c in term 0.
I20260928 08:30:25.681970 17805 raft_consensus.cc:2454] T 8093986aee6f4b2bb350eed151051d1b P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6848fd817df44775ab0088935a3c5f5c in term 0.
I20260928 08:30:25.682498 18017 leader_election.cc:302] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6848fd817df44775ab0088935a3c5f5c, cade9ccb06b54dec9030d39742e83380; no voters:
I20260928 08:30:25.682832 18226 raft_consensus.cc:2781] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:30:25.682986 18226 raft_consensus.cc:482] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:30:25.683082 18226 raft_consensus.cc:3037] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:25.685135 18226 raft_consensus.cc:504] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:25.685596 18226 leader_election.cc:288] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c [CANDIDATE]: Term 1 election: Requested vote from peers 4d6702619ea54e0e879bb172dc369ba3 (127.15.134.193:38513), cade9ccb06b54dec9030d39742e83380 (127.15.134.194:35915)
I20260928 08:30:25.686079 17805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8093986aee6f4b2bb350eed151051d1b" candidate_uuid: "6848fd817df44775ab0088935a3c5f5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d6702619ea54e0e879bb172dc369ba3"
I20260928 08:30:25.686091 17947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8093986aee6f4b2bb350eed151051d1b" candidate_uuid: "6848fd817df44775ab0088935a3c5f5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cade9ccb06b54dec9030d39742e83380"
I20260928 08:30:25.686305 17805 raft_consensus.cc:3037] T 8093986aee6f4b2bb350eed151051d1b P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:25.686327 17947 raft_consensus.cc:3037] T 8093986aee6f4b2bb350eed151051d1b P cade9ccb06b54dec9030d39742e83380 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:25.689237 17805 raft_consensus.cc:2454] T 8093986aee6f4b2bb350eed151051d1b P 4d6702619ea54e0e879bb172dc369ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6848fd817df44775ab0088935a3c5f5c in term 1.
I20260928 08:30:25.689433 17947 raft_consensus.cc:2454] T 8093986aee6f4b2bb350eed151051d1b P cade9ccb06b54dec9030d39742e83380 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6848fd817df44775ab0088935a3c5f5c in term 1.
I20260928 08:30:25.689637 18016 leader_election.cc:302] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d6702619ea54e0e879bb172dc369ba3, 6848fd817df44775ab0088935a3c5f5c; no voters:
I20260928 08:30:25.689906 18226 raft_consensus.cc:2781] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:25.690068 18226 raft_consensus.cc:686] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c [term 1 LEADER]: Becoming Leader. State: Replica: 6848fd817df44775ab0088935a3c5f5c, State: Running, Role: LEADER
I20260928 08:30:25.690341 18226 consensus_queue.cc:237] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:25.692956 17645 catalog_manager.cc:5495] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6848fd817df44775ab0088935a3c5f5c (127.15.134.195). New cstate: current_term: 1 leader_uuid: "6848fd817df44775ab0088935a3c5f5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } health_report { overall_health: HEALTHY } } }
I20260928 08:30:25.731455 18251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:25.726142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57072 (local address 127.15.134.195:40181)
0928 08:30:25.726826 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:25.726832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:25.726859 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:30:25.726906 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:25.726909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:25.726968 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:25.727098 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:30:25.727484 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:25.728271 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:25.729645 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:25.730102 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:25.730239 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:25.730687 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:30:25.730795 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:30:25.730899 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:25.731065 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 08:30:25.731137 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":410,"threads_started":1}
I20260928 08:30:25.746762 18251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:25.740711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:47657 (local address 127.15.134.195:40181)
0928 08:30:25.740871 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:25.740875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:25.741634 (+ 759us) server_negotiation.cc:407] Connection header received
0928 08:30:25.741839 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:25.741844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:25.741910 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:25.742001 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:30:25.742583 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:25.743325 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:25.744939 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:25.745420 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:25.745580 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:25.745953 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:30:25.746069 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:30:25.746167 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:25.746300 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 08:30:25.746385 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":53}
I20260928 08:30:25.746817 18254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:25.740696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:40181 (local address 127.15.134.195:47657)
0928 08:30:25.741528 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:25.741548 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:25.741720 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:25.742021 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:25.742025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:25.742035 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:30:25.742403 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:25.742411 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:25.743456 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:25.743462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:25.744801 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:25.744812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:25.745628 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:25.745634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:25.745766 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:25.745800 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:30:25.746168 (+ 368us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:30:25.746181 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 08:30:25.746285 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 08:30:25.746440 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":22,"thread_start_us":495,"threads_started":1}
I20260928 08:30:25.756419 17947 raft_consensus.cc:1261] T 8093986aee6f4b2bb350eed151051d1b P cade9ccb06b54dec9030d39742e83380 [term 1 FOLLOWER]: Refusing update from remote peer 6848fd817df44775ab0088935a3c5f5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:30:25.756438 17805 raft_consensus.cc:1261] T 8093986aee6f4b2bb350eed151051d1b P 4d6702619ea54e0e879bb172dc369ba3 [term 1 FOLLOWER]: Refusing update from remote peer 6848fd817df44775ab0088935a3c5f5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:30:25.757104 18217 consensus_queue.cc:1035] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:30:25.757527 18175 consensus_queue.cc:1035] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:30:25.823387 17645 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:30:25.823753 17645 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:30:25.827142 17645 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c: Sending DeleteTablet for 3 replicas of tablet 8093986aee6f4b2bb350eed151051d1b
I20260928 08:30:25.828029 17784 tablet_service.cc:1514] Processing DeleteTablet for tablet 8093986aee6f4b2bb350eed151051d1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:30:25 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:59082
I20260928 08:30:25.828052 17924 tablet_service.cc:1514] Processing DeleteTablet for tablet 8093986aee6f4b2bb350eed151051d1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:30:25 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:58430
I20260928 08:30:25.828243 18062 tablet_service.cc:1514] Processing DeleteTablet for tablet 8093986aee6f4b2bb350eed151051d1b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:30:25 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:57058
I20260928 08:30:25.828359 18231 tablet_replica.cc:330] T 8093986aee6f4b2bb350eed151051d1b P 4d6702619ea54e0e879bb172dc369ba3: stopping tablet replica
I20260928 08:30:25.828380 18232 tablet_replica.cc:330] T 8093986aee6f4b2bb350eed151051d1b P cade9ccb06b54dec9030d39742e83380: stopping tablet replica
I20260928 08:30:25.828577 18231 raft_consensus.cc:2229] T 8093986aee6f4b2bb350eed151051d1b P 4d6702619ea54e0e879bb172dc369ba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:30:25.828637 18232 raft_consensus.cc:2229] T 8093986aee6f4b2bb350eed151051d1b P cade9ccb06b54dec9030d39742e83380 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:30:25.828812 18231 raft_consensus.cc:2258] T 8093986aee6f4b2bb350eed151051d1b P 4d6702619ea54e0e879bb172dc369ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:30:25.828858 18232 raft_consensus.cc:2258] T 8093986aee6f4b2bb350eed151051d1b P cade9ccb06b54dec9030d39742e83380 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:30:25.829196 18233 tablet_replica.cc:330] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c: stopping tablet replica
I20260928 08:30:25.829703 18233 raft_consensus.cc:2229] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:30:25.830219 18231 ts_tablet_manager.cc:1905] T 8093986aee6f4b2bb350eed151051d1b P 4d6702619ea54e0e879bb172dc369ba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:30:25.830264 18233 raft_consensus.cc:2258] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:30:25.831380 18232 ts_tablet_manager.cc:1905] T 8093986aee6f4b2bb350eed151051d1b P cade9ccb06b54dec9030d39742e83380: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:30:25.831370 17645 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:30:25.833540 17645 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:30:25.835745 18233 ts_tablet_manager.cc:1905] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:30:25.839214 18232 ts_tablet_manager.cc:1918] T 8093986aee6f4b2bb350eed151051d1b P cade9ccb06b54dec9030d39742e83380: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:30:25.839419 18232 log.cc:1196] T 8093986aee6f4b2bb350eed151051d1b P cade9ccb06b54dec9030d39742e83380: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/8093986aee6f4b2bb350eed151051d1b
I20260928 08:30:25.840075 18232 ts_tablet_manager.cc:1939] T 8093986aee6f4b2bb350eed151051d1b P cade9ccb06b54dec9030d39742e83380: Deleting consensus metadata
I20260928 08:30:25.841122 18231 ts_tablet_manager.cc:1918] T 8093986aee6f4b2bb350eed151051d1b P 4d6702619ea54e0e879bb172dc369ba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:30:25.841280 18231 log.cc:1196] T 8093986aee6f4b2bb350eed151051d1b P 4d6702619ea54e0e879bb172dc369ba3: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-0/wal/wals/8093986aee6f4b2bb350eed151051d1b
I20260928 08:30:25.841343 17634 catalog_manager.cc:4841] TS cade9ccb06b54dec9030d39742e83380 (127.15.134.194:35915): tablet 8093986aee6f4b2bb350eed151051d1b (table test-table [id=de4bfa94f00d492ca5196e6331456e79]) successfully deleted
I20260928 08:30:25.841724 18231 ts_tablet_manager.cc:1939] T 8093986aee6f4b2bb350eed151051d1b P 4d6702619ea54e0e879bb172dc369ba3: Deleting consensus metadata
I20260928 08:30:25.843097 17633 catalog_manager.cc:4841] TS 4d6702619ea54e0e879bb172dc369ba3 (127.15.134.193:38513): tablet 8093986aee6f4b2bb350eed151051d1b (table test-table [id=de4bfa94f00d492ca5196e6331456e79]) successfully deleted
I20260928 08:30:25.843504 18233 ts_tablet_manager.cc:1918] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:30:25.843743 18233 log.cc:1196] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/8093986aee6f4b2bb350eed151051d1b
I20260928 08:30:25.844260 18233 ts_tablet_manager.cc:1939] T 8093986aee6f4b2bb350eed151051d1b P 6848fd817df44775ab0088935a3c5f5c: Deleting consensus metadata
I20260928 08:30:25.847322 18062 tablet_service.cc:1467] Processing CreateTablet for tablet 628f330070ae499b8029d1ba769127c4 (DEFAULT_TABLE table=test-table [id=9ba023908339447c93997f5be897e39d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:25.847299 17784 tablet_service.cc:1467] Processing CreateTablet for tablet 628f330070ae499b8029d1ba769127c4 (DEFAULT_TABLE table=test-table [id=9ba023908339447c93997f5be897e39d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:25.847419 17924 tablet_service.cc:1467] Processing CreateTablet for tablet 628f330070ae499b8029d1ba769127c4 (DEFAULT_TABLE table=test-table [id=9ba023908339447c93997f5be897e39d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:25.847729 17784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 628f330070ae499b8029d1ba769127c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:25.847819 18062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 628f330070ae499b8029d1ba769127c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:25.847894 17924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 628f330070ae499b8029d1ba769127c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:25.854419 18242 tablet_bootstrap.cc:492] T 628f330070ae499b8029d1ba769127c4 P cade9ccb06b54dec9030d39742e83380: Bootstrap starting.
I20260928 08:30:25.855403 18243 tablet_bootstrap.cc:492] T 628f330070ae499b8029d1ba769127c4 P 6848fd817df44775ab0088935a3c5f5c: Bootstrap starting.
I20260928 08:30:25.856210 18242 tablet_bootstrap.cc:654] T 628f330070ae499b8029d1ba769127c4 P cade9ccb06b54dec9030d39742e83380: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:25.857292 18243 tablet_bootstrap.cc:654] T 628f330070ae499b8029d1ba769127c4 P 6848fd817df44775ab0088935a3c5f5c: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:25.857486 17632 catalog_manager.cc:4841] TS 6848fd817df44775ab0088935a3c5f5c (127.15.134.195:40181): tablet 8093986aee6f4b2bb350eed151051d1b (table test-table [id=de4bfa94f00d492ca5196e6331456e79]) successfully deleted
I20260928 08:30:25.859201 18244 tablet_bootstrap.cc:492] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3: Bootstrap starting.
I20260928 08:30:25.859771 18242 tablet_bootstrap.cc:492] T 628f330070ae499b8029d1ba769127c4 P cade9ccb06b54dec9030d39742e83380: No bootstrap required, opened a new log
I20260928 08:30:25.859973 18242 ts_tablet_manager.cc:1397] T 628f330070ae499b8029d1ba769127c4 P cade9ccb06b54dec9030d39742e83380: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260928 08:30:25.860986 18242 raft_consensus.cc:348] T 628f330070ae499b8029d1ba769127c4 P cade9ccb06b54dec9030d39742e83380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:25.861354 18242 raft_consensus.cc:374] T 628f330070ae499b8029d1ba769127c4 P cade9ccb06b54dec9030d39742e83380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:25.861477 18242 raft_consensus.cc:729] T 628f330070ae499b8029d1ba769127c4 P cade9ccb06b54dec9030d39742e83380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cade9ccb06b54dec9030d39742e83380, State: Initialized, Role: FOLLOWER
I20260928 08:30:25.861759 18242 consensus_queue.cc:260] T 628f330070ae499b8029d1ba769127c4 P cade9ccb06b54dec9030d39742e83380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:25.862370 18242 ts_tablet_manager.cc:1428] T 628f330070ae499b8029d1ba769127c4 P cade9ccb06b54dec9030d39742e83380: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:30:25.862699 18244 tablet_bootstrap.cc:654] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:25.865700 18244 tablet_bootstrap.cc:492] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3: No bootstrap required, opened a new log
I20260928 08:30:25.865877 18243 tablet_bootstrap.cc:492] T 628f330070ae499b8029d1ba769127c4 P 6848fd817df44775ab0088935a3c5f5c: No bootstrap required, opened a new log
I20260928 08:30:25.865876 18244 ts_tablet_manager.cc:1397] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260928 08:30:25.866122 18243 ts_tablet_manager.cc:1397] T 628f330070ae499b8029d1ba769127c4 P 6848fd817df44775ab0088935a3c5f5c: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260928 08:30:25.866770 18244 raft_consensus.cc:348] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:25.867098 18244 raft_consensus.cc:374] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:25.867187 18243 raft_consensus.cc:348] T 628f330070ae499b8029d1ba769127c4 P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:25.867298 18244 raft_consensus.cc:729] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d6702619ea54e0e879bb172dc369ba3, State: Initialized, Role: FOLLOWER
I20260928 08:30:25.867451 18243 raft_consensus.cc:374] T 628f330070ae499b8029d1ba769127c4 P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:25.867548 18243 raft_consensus.cc:729] T 628f330070ae499b8029d1ba769127c4 P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6848fd817df44775ab0088935a3c5f5c, State: Initialized, Role: FOLLOWER
I20260928 08:30:25.867724 18244 consensus_queue.cc:260] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:25.867779 18243 consensus_queue.cc:260] T 628f330070ae499b8029d1ba769127c4 P 6848fd817df44775ab0088935a3c5f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:25.868427 18244 ts_tablet_manager.cc:1428] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 08:30:25.869436 18243 ts_tablet_manager.cc:1428] T 628f330070ae499b8029d1ba769127c4 P 6848fd817df44775ab0088935a3c5f5c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 08:30:25.952244 18222 raft_consensus.cc:482] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:30:25.952487 18222 raft_consensus.cc:504] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:25.953336 18222 leader_election.cc:288] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cade9ccb06b54dec9030d39742e83380 (127.15.134.194:35915), 6848fd817df44775ab0088935a3c5f5c (127.15.134.195:40181)
I20260928 08:30:25.954109 17946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628f330070ae499b8029d1ba769127c4" candidate_uuid: "4d6702619ea54e0e879bb172dc369ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cade9ccb06b54dec9030d39742e83380" is_pre_election: true
I20260928 08:30:25.954171 18087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628f330070ae499b8029d1ba769127c4" candidate_uuid: "4d6702619ea54e0e879bb172dc369ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6848fd817df44775ab0088935a3c5f5c" is_pre_election: true
I20260928 08:30:25.954419 17946 raft_consensus.cc:2454] T 628f330070ae499b8029d1ba769127c4 P cade9ccb06b54dec9030d39742e83380 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d6702619ea54e0e879bb172dc369ba3 in term 0.
I20260928 08:30:25.954487 18087 raft_consensus.cc:2454] T 628f330070ae499b8029d1ba769127c4 P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d6702619ea54e0e879bb172dc369ba3 in term 0.
I20260928 08:30:25.954929 17737 leader_election.cc:302] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d6702619ea54e0e879bb172dc369ba3, cade9ccb06b54dec9030d39742e83380; no voters:
I20260928 08:30:25.955302 18222 raft_consensus.cc:2781] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:30:25.955416 18222 raft_consensus.cc:482] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:30:25.955488 18222 raft_consensus.cc:3037] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:25.959064 18222 raft_consensus.cc:504] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:25.960244 17946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628f330070ae499b8029d1ba769127c4" candidate_uuid: "4d6702619ea54e0e879bb172dc369ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cade9ccb06b54dec9030d39742e83380"
I20260928 08:30:25.960269 18087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "628f330070ae499b8029d1ba769127c4" candidate_uuid: "4d6702619ea54e0e879bb172dc369ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6848fd817df44775ab0088935a3c5f5c"
I20260928 08:30:25.960479 18087 raft_consensus.cc:3037] T 628f330070ae499b8029d1ba769127c4 P 6848fd817df44775ab0088935a3c5f5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:25.960500 17946 raft_consensus.cc:3037] T 628f330070ae499b8029d1ba769127c4 P cade9ccb06b54dec9030d39742e83380 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:25.964309 17946 raft_consensus.cc:2454] T 628f330070ae499b8029d1ba769127c4 P cade9ccb06b54dec9030d39742e83380 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d6702619ea54e0e879bb172dc369ba3 in term 1.
I20260928 08:30:25.965047 17737 leader_election.cc:302] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d6702619ea54e0e879bb172dc369ba3, cade9ccb06b54dec9030d39742e83380; no voters:
I20260928 08:30:25.965041 18222 leader_election.cc:288] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3 [CANDIDATE]: Term 1 election: Requested vote from peers cade9ccb06b54dec9030d39742e83380 (127.15.134.194:35915), 6848fd817df44775ab0088935a3c5f5c (127.15.134.195:40181)
I20260928 08:30:25.965257 18087 raft_consensus.cc:2454] T 628f330070ae499b8029d1ba769127c4 P 6848fd817df44775ab0088935a3c5f5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d6702619ea54e0e879bb172dc369ba3 in term 1.
I20260928 08:30:25.965463 18222 raft_consensus.cc:2781] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:25.965778 18222 raft_consensus.cc:686] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3 [term 1 LEADER]: Becoming Leader. State: Replica: 4d6702619ea54e0e879bb172dc369ba3, State: Running, Role: LEADER
I20260928 08:30:25.966233 18222 consensus_queue.cc:237] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } }
I20260928 08:30:25.970059 17645 catalog_manager.cc:5495] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d6702619ea54e0e879bb172dc369ba3 (127.15.134.193). New cstate: current_term: 1 leader_uuid: "4d6702619ea54e0e879bb172dc369ba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6702619ea54e0e879bb172dc369ba3" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 } health_report { overall_health: UNKNOWN } } }
I20260928 08:30:26.002503 18262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:25.995957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59086 (local address 127.15.134.193:38513)
0928 08:30:25.996707 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:25.996711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:25.996754 (+ 43us) server_negotiation.cc:407] Connection header received
0928 08:30:25.996813 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:25.996817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:25.996878 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:25.996961 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:30:25.997383 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:25.998272 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:26.000447 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:26.000956 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:26.001062 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:26.001702 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:30:26.001816 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:30:26.001884 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:26.002080 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 08:30:26.002142 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":320,"threads_started":1}
I20260928 08:30:26.012778 17946 raft_consensus.cc:1261] T 628f330070ae499b8029d1ba769127c4 P cade9ccb06b54dec9030d39742e83380 [term 1 FOLLOWER]: Refusing update from remote peer 4d6702619ea54e0e879bb172dc369ba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:30:26.012960 18087 raft_consensus.cc:1261] T 628f330070ae499b8029d1ba769127c4 P 6848fd817df44775ab0088935a3c5f5c [term 1 FOLLOWER]: Refusing update from remote peer 4d6702619ea54e0e879bb172dc369ba3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:30:26.013397 18164 consensus_queue.cc:1035] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cade9ccb06b54dec9030d39742e83380" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:30:26.013664 18222 consensus_queue.cc:1035] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6848fd817df44775ab0088935a3c5f5c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 40181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 08:30:26.044214 18214 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 08:30:27.637936 17645 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:30:27.638268 17645 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:30:27.643087 17645 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42c341dda8ec4da0a76526e04c0d1f8c: Sending DeleteTablet for 3 replicas of tablet 628f330070ae499b8029d1ba769127c4
I20260928 08:30:27.644052 17924 tablet_service.cc:1514] Processing DeleteTablet for tablet 628f330070ae499b8029d1ba769127c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:30:27 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:58430
I20260928 08:30:27.643970 17784 tablet_service.cc:1514] Processing DeleteTablet for tablet 628f330070ae499b8029d1ba769127c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:30:27 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:59082
I20260928 08:30:27.644193 18062 tablet_service.cc:1514] Processing DeleteTablet for tablet 628f330070ae499b8029d1ba769127c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:30:27 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:57058
I20260928 08:30:27.645269 18290 tablet_replica.cc:330] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3: stopping tablet replica
I20260928 08:30:27.645587 18290 raft_consensus.cc:2229] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:30:27.645664 18291 tablet_replica.cc:330] T 628f330070ae499b8029d1ba769127c4 P 6848fd817df44775ab0088935a3c5f5c: stopping tablet replica
I20260928 08:30:27.646314 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 17718
I20260928 08:30:27.646584 18291 raft_consensus.cc:2229] T 628f330070ae499b8029d1ba769127c4 P 6848fd817df44775ab0088935a3c5f5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:30:27.646962 18291 raft_consensus.cc:2258] T 628f330070ae499b8029d1ba769127c4 P 6848fd817df44775ab0088935a3c5f5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:30:27.647074 18290 raft_consensus.cc:2258] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:30:27.648078 18290 ts_tablet_manager.cc:1905] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:30:27.648224 18291 ts_tablet_manager.cc:1905] T 628f330070ae499b8029d1ba769127c4 P 6848fd817df44775ab0088935a3c5f5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:30:27.654289 18290 ts_tablet_manager.cc:1918] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:30:27.654551 18290 log.cc:1196] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-0/wal/wals/628f330070ae499b8029d1ba769127c4
I20260928 08:30:27.654838 18289 tablet_replica.cc:330] T 628f330070ae499b8029d1ba769127c4 P cade9ccb06b54dec9030d39742e83380: stopping tablet replica
I20260928 08:30:27.655076 18289 raft_consensus.cc:2229] T 628f330070ae499b8029d1ba769127c4 P cade9ccb06b54dec9030d39742e83380 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:30:27.655136 18290 ts_tablet_manager.cc:1939] T 628f330070ae499b8029d1ba769127c4 P 4d6702619ea54e0e879bb172dc369ba3: Deleting consensus metadata
I20260928 08:30:27.655431 18289 raft_consensus.cc:2258] T 628f330070ae499b8029d1ba769127c4 P cade9ccb06b54dec9030d39742e83380 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:30:27.656325 17633 catalog_manager.cc:4841] TS 4d6702619ea54e0e879bb172dc369ba3 (127.15.134.193:38513): tablet 628f330070ae499b8029d1ba769127c4 (table test-table [id=9ba023908339447c93997f5be897e39d]) successfully deleted
I20260928 08:30:27.657059 18289 ts_tablet_manager.cc:1905] T 628f330070ae499b8029d1ba769127c4 P cade9ccb06b54dec9030d39742e83380: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:30:27.657940 18291 ts_tablet_manager.cc:1918] T 628f330070ae499b8029d1ba769127c4 P 6848fd817df44775ab0088935a3c5f5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:30:27.658214 18291 log.cc:1196] T 628f330070ae499b8029d1ba769127c4 P 6848fd817df44775ab0088935a3c5f5c: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/628f330070ae499b8029d1ba769127c4
I20260928 08:30:27.658953 18291 ts_tablet_manager.cc:1939] T 628f330070ae499b8029d1ba769127c4 P 6848fd817df44775ab0088935a3c5f5c: Deleting consensus metadata
I20260928 08:30:27.660128 17632 catalog_manager.cc:4841] TS 6848fd817df44775ab0088935a3c5f5c (127.15.134.195:40181): tablet 628f330070ae499b8029d1ba769127c4 (table test-table [id=9ba023908339447c93997f5be897e39d]) successfully deleted
I20260928 08:30:27.663440 18289 ts_tablet_manager.cc:1918] T 628f330070ae499b8029d1ba769127c4 P cade9ccb06b54dec9030d39742e83380: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:30:27.663579 18289 log.cc:1196] T 628f330070ae499b8029d1ba769127c4 P cade9ccb06b54dec9030d39742e83380: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/628f330070ae499b8029d1ba769127c4
I20260928 08:30:27.664019 18289 ts_tablet_manager.cc:1939] T 628f330070ae499b8029d1ba769127c4 P cade9ccb06b54dec9030d39742e83380: Deleting consensus metadata
I20260928 08:30:27.665127 17634 catalog_manager.cc:4841] TS cade9ccb06b54dec9030d39742e83380 (127.15.134.194:35915): tablet 628f330070ae499b8029d1ba769127c4 (table test-table [id=9ba023908339447c93997f5be897e39d]) successfully deleted
Sep 28 08:30:27 dist-test-slave-qp2n krb5kdc[17602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584225, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.134.193@KRBTEST.COM
I20260928 08:30:27.676530 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:27.647953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47862 (local address 127.15.134.193:38513)
0928 08:30:27.654612 (+ 6659us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:27.654620 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:27.654648 (+ 28us) server_negotiation.cc:407] Connection header received
0928 08:30:27.654729 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:27.654737 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:27.654852 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:27.656802 (+ 1950us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:27.657108 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:27.658205 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:27.659806 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:27.660426 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:27.660583 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:27.669713 (+ 9130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:27.669732 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:27.669751 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:27.669803 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:27.673390 (+ 3587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:27.674167 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:27.674176 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:27.674179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:27.674341 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:27.674761 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:27.674766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:27.674771 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:27.675252 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:27.675571 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:27.675907 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 08:30:27.676158 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6453,"thread_start_us":6344,"threads_started":1}
W20260928 08:30:27.677281 17849 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47862
W20260928 08:30:27.677904 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 17718: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:30:27.678112 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 17718
I20260928 08:30:27.702867 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 17858
Sep 28 08:30:27 dist-test-slave-qp2n krb5kdc[17602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584225, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.134.194@KRBTEST.COM
I20260928 08:30:27.719035 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:27.703576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42330 (local address 127.15.134.194:35915)
0928 08:30:27.704153 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:27.704157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:27.704172 (+ 15us) server_negotiation.cc:407] Connection header received
0928 08:30:27.704225 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:27.704229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:27.704299 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:27.704422 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:27.705784 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:27.706533 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:27.707812 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:27.708222 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:27.708359 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:27.712899 (+ 4540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:27.712913 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:27.712929 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:27.712972 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:27.716016 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:27.716778 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:27.716785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:27.716788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:27.716929 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:27.717344 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:27.717348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:27.717350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:27.717801 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:27.718146 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:27.718441 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 08:30:27.718663 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":320,"threads_started":1}
W20260928 08:30:27.719780 17989 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42330
W20260928 08:30:27.720394 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 17858: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:30:27.720530 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 17858
I20260928 08:30:27.739694 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 17998
Sep 28 08:30:27 dist-test-slave-qp2n krb5kdc[17602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584225, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.15.134.195@KRBTEST.COM
I20260928 08:30:27.755841 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:27.740463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46072 (local address 127.15.134.195:40181)
0928 08:30:27.741068 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:27.741073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:27.741087 (+ 14us) server_negotiation.cc:407] Connection header received
0928 08:30:27.741137 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:27.741149 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:27.741206 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:27.741332 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:27.742646 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:27.743426 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:27.744703 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:27.745101 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:27.745244 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:27.749657 (+ 4413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:27.749673 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:27.749690 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:27.749724 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:27.752432 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:27.753339 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:27.753357 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:27.753360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:27.753532 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:27.753989 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:27.754001 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:27.754003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:27.754432 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:27.754758 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:27.755205 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 08:30:27.755458 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":341,"threads_started":1}
W20260928 08:30:27.756579 18128 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46072
W20260928 08:30:27.757310 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 17998: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:30:27.757547 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 17998
I20260928 08:30:27.784096 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 17615
I20260928 08:30:27.785425 17687 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:27.903219 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 17615
2026-09-28T08:30:27Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8470 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:30:28 dist-test-slave-qp2n krb5kdc[18299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:30:28 dist-test-slave-qp2n krb5kdc[18299](info): set up 2 sockets
Sep 28 08:30:28 dist-test-slave-qp2n krb5kdc[18299](info): commencing operation
krb5kdc: starting...
W20260928 08:30:29.014712 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:30:29 dist-test-slave-qp2n krb5kdc[18299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:30:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:30:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:30:29.214740 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:33189
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:35687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:33189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:29.547567 18312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:29.547856 18312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:29.547945 18312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:29.555172 18312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:30:29.555292 18312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:29.555338 18312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:29.555388 18312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:30:29.555433 18312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:30:29.565368 18312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:35687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:33189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:33189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:29.566735 18312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:29.569761 18312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:29.588555 18317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:29.588651 18318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:29.588527 18320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:29.589422 18312 server_base.cc:1034] running on GCE node
I20260928 08:30:29.590329 18312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:29.591956 18312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:29.593214 18312 hybrid_clock.cc:648] HybridClock initialized: now 1790584229593159 us; error 49 us; skew 500 ppm
I20260928 08:30:29.593655 18312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:29 dist-test-slave-qp2n krb5kdc[18299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584229, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:29.597150 18312 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:30:29.598866 18312 webserver.cc:460] Webserver started at http://127.15.134.254:36207/ using document root <none> and password file <none>
I20260928 08:30:29.599668 18312 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:29.599841 18312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:29.600189 18312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:29.603494 18312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "f7b052c47f3c4067bae90cf9f0e04465"
format_stamp: "Formatted at 2026-09-28 08:30:29 on dist-test-slave-qp2n"
server_key: "609c169cf775580ebbc76d2514147807"
server_key_iv: "da22bf86ddfeb2b077e7f8773b8d4e86"
server_key_version: "encryptionkey@0"
I20260928 08:30:29.604331 18312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "f7b052c47f3c4067bae90cf9f0e04465"
format_stamp: "Formatted at 2026-09-28 08:30:29 on dist-test-slave-qp2n"
server_key: "609c169cf775580ebbc76d2514147807"
server_key_iv: "da22bf86ddfeb2b077e7f8773b8d4e86"
server_key_version: "encryptionkey@0"
I20260928 08:30:29.610584 18312 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 08:30:29.614782 18327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:29.616478 18312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:30:29.616741 18312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "f7b052c47f3c4067bae90cf9f0e04465"
format_stamp: "Formatted at 2026-09-28 08:30:29 on dist-test-slave-qp2n"
server_key: "609c169cf775580ebbc76d2514147807"
server_key_iv: "da22bf86ddfeb2b077e7f8773b8d4e86"
server_key_version: "encryptionkey@0"
I20260928 08:30:29.616992 18312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:29.646077 18312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:29.650844 18312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:29.651226 18312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:29.671932 18312 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:33189
I20260928 08:30:29.671983 18378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:33189 every 8 connection(s)
I20260928 08:30:29.673489 18312 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:30:29.674180 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 18312
I20260928 08:30:29.674459 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:30:29.675235 15899 external_mini_cluster.cc:1428] Setting key 4ab63cb6dd5f722491ed470f3e3e522d
I20260928 08:30:29.678371 18379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 08:30:29 dist-test-slave-qp2n krb5kdc[18299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:29.689414 18379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7b052c47f3c4067bae90cf9f0e04465: Bootstrap starting.
I20260928 08:30:29.692992 18379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7b052c47f3c4067bae90cf9f0e04465: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:29.694082 18379 log.cc:824] T 00000000000000000000000000000000 P f7b052c47f3c4067bae90cf9f0e04465: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:29.697392 18379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7b052c47f3c4067bae90cf9f0e04465: No bootstrap required, opened a new log
I20260928 08:30:29.697983 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:29.677410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47226 (local address 127.15.134.254:33189)
0928 08:30:29.678365 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:29.678379 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:29.678421 (+ 42us) server_negotiation.cc:407] Connection header received
0928 08:30:29.680163 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:29.680185 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:29.680580 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:29.681078 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:29.682523 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:29.683865 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:29.685182 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:29.685909 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:29.686168 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:29.690265 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:29.690273 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:29.690288 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:29.690329 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:29.693585 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:29.694313 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:29.694320 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:29.694326 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:29.694490 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:29.694996 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:29.695004 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:29.695010 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:29.695679 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:29.695903 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:29.696422 (+ 519us) server_negotiation.cc:299] Negotiation successful
0928 08:30:29.696898 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":403,"threads_started":1}
I20260928 08:30:29.705076 18379 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7b052c47f3c4067bae90cf9f0e04465 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b052c47f3c4067bae90cf9f0e04465" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 33189 } }
I20260928 08:30:29.705461 18379 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7b052c47f3c4067bae90cf9f0e04465 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:29.705571 18379 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7b052c47f3c4067bae90cf9f0e04465 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7b052c47f3c4067bae90cf9f0e04465, State: Initialized, Role: FOLLOWER
I20260928 08:30:29.706166 18379 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7b052c47f3c4067bae90cf9f0e04465 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b052c47f3c4067bae90cf9f0e04465" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 33189 } }
I20260928 08:30:29.706421 18379 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7b052c47f3c4067bae90cf9f0e04465 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:30:29.706565 18379 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7b052c47f3c4067bae90cf9f0e04465 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:30:29.706761 18379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7b052c47f3c4067bae90cf9f0e04465 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:29.709196 18379 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7b052c47f3c4067bae90cf9f0e04465 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b052c47f3c4067bae90cf9f0e04465" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 33189 } }
I20260928 08:30:29.709630 18379 leader_election.cc:302] T 00000000000000000000000000000000 P f7b052c47f3c4067bae90cf9f0e04465 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7b052c47f3c4067bae90cf9f0e04465; no voters:
I20260928 08:30:29.710251 18379 leader_election.cc:288] T 00000000000000000000000000000000 P f7b052c47f3c4067bae90cf9f0e04465 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:30:29.710414 18384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7b052c47f3c4067bae90cf9f0e04465 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:29.711090 18384 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7b052c47f3c4067bae90cf9f0e04465 [term 1 LEADER]: Becoming Leader. State: Replica: f7b052c47f3c4067bae90cf9f0e04465, State: Running, Role: LEADER
I20260928 08:30:29.711612 18384 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7b052c47f3c4067bae90cf9f0e04465 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b052c47f3c4067bae90cf9f0e04465" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 33189 } }
I20260928 08:30:29.712314 18379 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7b052c47f3c4067bae90cf9f0e04465 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:30:29.714731 18386 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b052c47f3c4067bae90cf9f0e04465 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7b052c47f3c4067bae90cf9f0e04465. Latest consensus state: current_term: 1 leader_uuid: "f7b052c47f3c4067bae90cf9f0e04465" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b052c47f3c4067bae90cf9f0e04465" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 33189 } } }
I20260928 08:30:29.715126 18386 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b052c47f3c4067bae90cf9f0e04465 [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:29.714962 18385 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b052c47f3c4067bae90cf9f0e04465 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7b052c47f3c4067bae90cf9f0e04465" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b052c47f3c4067bae90cf9f0e04465" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 33189 } } }
I20260928 08:30:29.715337 18385 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b052c47f3c4067bae90cf9f0e04465 [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:29.717643 18391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:30:29.723513 18391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:30:29.732898 18391 catalog_manager.cc:1347] Generated new cluster ID: 6e7cec8223e54dd788bd0242a5c4d9b6
I20260928 08:30:29.733106 18391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:30:29.756691 18391 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:30:29.758152 18391 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:30:29.772505 18391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7b052c47f3c4067bae90cf9f0e04465: Generated new TSK 0
I20260928 08:30:29.773567 18391 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.134.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
I20260928 08:30:29.832787 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:0
--local_ip_for_outbound_sockets=127.15.134.193
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:33189
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:35687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:30.164810 18405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:30.165125 18405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:30.165292 18405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:30.172962 18405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:30.173079 18405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:30.173270 18405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:30:30.183446 18405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:35687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:33189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:30.184892 18405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:30.186259 18405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:30.200042 18410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:30.200042 18413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:30.200083 18411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:30.201390 18405 server_base.cc:1034] running on GCE node
I20260928 08:30:30.202070 18405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:30.203117 18405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:30.204439 18405 hybrid_clock.cc:648] HybridClock initialized: now 1790584230204364 us; error 50 us; skew 500 ppm
I20260928 08:30:30.204846 18405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:30 dist-test-slave-qp2n krb5kdc[18299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584230, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:30.208195 18405 init.cc:377] Logged in from keytab as kudu/127.15.134.193@KRBTEST.COM (short username kudu)
I20260928 08:30:30.209839 18405 webserver.cc:460] Webserver started at http://127.15.134.193:42325/ using document root <none> and password file <none>
I20260928 08:30:30.210593 18405 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:30.210762 18405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:30.211122 18405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:30.214738 18405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/data/instance:
uuid: "e0d7b868dd93405b969bc73b5f3c49da"
format_stamp: "Formatted at 2026-09-28 08:30:30 on dist-test-slave-qp2n"
server_key: "c3bf6238b0e67531ec1d8a78958f9a80"
server_key_iv: "33180bbe0a8204169ced37c28e1260fa"
server_key_version: "encryptionkey@0"
I20260928 08:30:30.215651 18405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance:
uuid: "e0d7b868dd93405b969bc73b5f3c49da"
format_stamp: "Formatted at 2026-09-28 08:30:30 on dist-test-slave-qp2n"
server_key: "c3bf6238b0e67531ec1d8a78958f9a80"
server_key_iv: "33180bbe0a8204169ced37c28e1260fa"
server_key_version: "encryptionkey@0"
I20260928 08:30:30.221787 18405 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 08:30:30.225503 18420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:30.226871 18405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:30:30.227044 18405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "e0d7b868dd93405b969bc73b5f3c49da"
format_stamp: "Formatted at 2026-09-28 08:30:30 on dist-test-slave-qp2n"
server_key: "c3bf6238b0e67531ec1d8a78958f9a80"
server_key_iv: "33180bbe0a8204169ced37c28e1260fa"
server_key_version: "encryptionkey@0"
I20260928 08:30:30.227242 18405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:30.245105 18405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:30.250133 18405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:30.250532 18405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:30.251797 18405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:30:30.253539 18405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:30.253691 18405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:30.253877 18405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:30.253968 18405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:30.291402 18405 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:40147
I20260928 08:30:30.291448 18532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:40147 every 8 connection(s)
I20260928 08:30:30.292824 18405 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:30:30.294487 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 18405
I20260928 08:30:30.294720 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance
I20260928 08:30:30.295488 15899 external_mini_cluster.cc:1428] Setting key e99548129acc5f1bc637a052bfa5b0aa
Sep 28 08:30:30 dist-test-slave-qp2n krb5kdc[18299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584230, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.134.194@KRBTEST.COM; defaulting to no policy
I20260928 08:30:30.316764 18535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:30.295728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:43843 (local address 127.15.134.254:33189)
0928 08:30:30.296237 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:30.296245 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:30.297636 (+ 1391us) server_negotiation.cc:407] Connection header received
0928 08:30:30.299519 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:30.299527 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:30.299618 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:30.299735 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:30.301460 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:30.302512 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:30.304654 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:30.305382 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:30.305542 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:30.309438 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:30.309455 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:30.309463 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:30.309512 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:30.313426 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:30.314245 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:30.314252 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:30.314257 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:30.314413 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:30.314880 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:30.314885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:30.314889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:30.315314 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:30.315470 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:30.316266 (+ 796us) server_negotiation.cc:299] Negotiation successful
0928 08:30:30.316436 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":258,"threads_started":1}
I20260928 08:30:30.317899 18536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:30.296550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:33189 (local address 127.15.134.193:43843)
0928 08:30:30.297459 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:30.297514 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:30.299231 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:30.299972 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:30.299984 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:30.300562 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:30.301264 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:30.301282 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:30.302690 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:30.302697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:30.304469 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:30.304481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:30.305615 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:30.305620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:30.305824 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:30.306541 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:30.306567 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:30.309171 (+ 2604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:30.313580 (+ 4409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:30.313588 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:30.313603 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:30.314091 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:30.314546 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:30.314550 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:30.314552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:30.314745 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:30.315476 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:30.315483 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:30.315835 (+ 352us) client_negotiation.cc:769] Sending connection context
0928 08:30:30.316075 (+ 240us) client_negotiation.cc:240] Negotiation successful
0928 08:30:30.316571 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":348,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.194@KRBTEST.COM" created.
I20260928 08:30:30.319756 18533 heartbeater.cc:347] Connected to a master server at 127.15.134.254:33189
I20260928 08:30:30.320184 18533 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:30.321168 18533 heartbeater.cc:511] Master 127.15.134.254:33189 requested a full tablet report, sending...
I20260928 08:30:30.323891 18344 ts_manager.cc:194] Registered new tserver with Master: e0d7b868dd93405b969bc73b5f3c49da (127.15.134.193:40147)
I20260928 08:30:30.325503 18344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.193@KRBTEST.COM'} at 127.15.134.193:43843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
I20260928 08:30:30.344349 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:0
--local_ip_for_outbound_sockets=127.15.134.194
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:33189
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:35687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:30.683626 18539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:30.683943 18539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:30.684110 18539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:30.691727 18539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:30.691890 18539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:30.692068 18539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:30:30.702732 18539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:35687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:33189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:30.704187 18539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:30.705659 18539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:30.720925 18544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:30.720810 18547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:30.720850 18545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:30.721747 18539 server_base.cc:1034] running on GCE node
I20260928 08:30:30.722496 18539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:30.723526 18539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:30.724725 18539 hybrid_clock.cc:648] HybridClock initialized: now 1790584230724671 us; error 64 us; skew 500 ppm
I20260928 08:30:30.725106 18539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:30 dist-test-slave-qp2n krb5kdc[18299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584230, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:30.728469 18539 init.cc:377] Logged in from keytab as kudu/127.15.134.194@KRBTEST.COM (short username kudu)
I20260928 08:30:30.730079 18539 webserver.cc:460] Webserver started at http://127.15.134.194:37221/ using document root <none> and password file <none>
I20260928 08:30:30.730827 18539 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:30.730971 18539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:30.731276 18539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:30.734301 18539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/data/instance:
uuid: "78864ec996a543e581725ad1143ef49b"
format_stamp: "Formatted at 2026-09-28 08:30:30 on dist-test-slave-qp2n"
server_key: "5408eae21e1e07a9d4dba3ad91c88c6f"
server_key_iv: "b14d7ed2291130d5f83c9bd7afe952e2"
server_key_version: "encryptionkey@0"
I20260928 08:30:30.735082 18539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance:
uuid: "78864ec996a543e581725ad1143ef49b"
format_stamp: "Formatted at 2026-09-28 08:30:30 on dist-test-slave-qp2n"
server_key: "5408eae21e1e07a9d4dba3ad91c88c6f"
server_key_iv: "b14d7ed2291130d5f83c9bd7afe952e2"
server_key_version: "encryptionkey@0"
I20260928 08:30:30.740293 18539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 08:30:30.744234 18554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:30.746016 18539 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 08:30:30.746232 18539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "78864ec996a543e581725ad1143ef49b"
format_stamp: "Formatted at 2026-09-28 08:30:30 on dist-test-slave-qp2n"
server_key: "5408eae21e1e07a9d4dba3ad91c88c6f"
server_key_iv: "b14d7ed2291130d5f83c9bd7afe952e2"
server_key_version: "encryptionkey@0"
I20260928 08:30:30.746455 18539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:30.765375 18539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:30.770126 18539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:30.770437 18539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:30.771442 18539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:30:30.773078 18539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:30.773178 18539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:30.773350 18539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:30.773427 18539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:30.809130 18539 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:40827
I20260928 08:30:30.809213 18666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:40827 every 8 connection(s)
I20260928 08:30:30.810410 18539 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:30:30.813939 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 18539
I20260928 08:30:30.814145 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance
I20260928 08:30:30.814594 15899 external_mini_cluster.cc:1428] Setting key 7e22c0c834342d83fef18987bbe2a645
Sep 28 08:30:30 dist-test-slave-qp2n krb5kdc[18299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584230, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:30.832304 18535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:30.813250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:59493 (local address 127.15.134.254:33189)
0928 08:30:30.813436 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:30.813443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:30.814724 (+ 1281us) server_negotiation.cc:407] Connection header received
0928 08:30:30.817170 (+ 2446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:30.817178 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:30.817267 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:30.817373 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:30.819219 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:30.820397 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:30.822215 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:30.822659 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:30.822798 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:30.826377 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:30.826382 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:30.826386 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:30.826420 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:30.829153 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:30.829904 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:30.829917 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:30.829920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:30.830044 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:30.830512 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:30.830517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:30.830519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:30.830802 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:30.831023 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:30.831702 (+ 679us) server_negotiation.cc:299] Negotiation successful
0928 08:30:30.831929 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.15.134.195@KRBTEST.COM; defaulting to no policy
I20260928 08:30:30.833467 18669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:30.813726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:33189 (local address 127.15.134.194:59493)
0928 08:30:30.814541 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:30.814606 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:30.816890 (+ 2284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:30.817647 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:30.817655 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:30.818325 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:30.819044 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:30.819063 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:30.820646 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:30.820652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:30.822064 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:30.822077 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:30.822911 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:30.822918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:30.823166 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:30.823877 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:30.823905 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:30.826127 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:30.829311 (+ 3184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:30.829321 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:30.829339 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:30.829770 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:30.830157 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:30.830163 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:30.830167 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:30.830358 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:30.830960 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:30.830970 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:30.831336 (+ 366us) client_negotiation.cc:769] Sending connection context
0928 08:30:30.831624 (+ 288us) client_negotiation.cc:240] Negotiation successful
0928 08:30:30.832191 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":334,"threads_started":1}
I20260928 08:30:30.835018 18667 heartbeater.cc:347] Connected to a master server at 127.15.134.254:33189
I20260928 08:30:30.835348 18667 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.195@KRBTEST.COM" created.
I20260928 08:30:30.836202 18667 heartbeater.cc:511] Master 127.15.134.254:33189 requested a full tablet report, sending...
I20260928 08:30:30.837736 18344 ts_manager.cc:194] Registered new tserver with Master: 78864ec996a543e581725ad1143ef49b (127.15.134.194:40827)
I20260928 08:30:30.838594 18344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.194@KRBTEST.COM'} at 127.15.134.194:59493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
I20260928 08:30:30.859782 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:0
--local_ip_for_outbound_sockets=127.15.134.195
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:33189
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:35687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:31.194084 18672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:31.194375 18672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:31.194540 18672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:31.201800 18672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:31.201898 18672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:31.202070 18672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:30:31.211611 18672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:35687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:33189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:31.212953 18672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:31.214299 18672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:31.228328 18677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:31.228313 18678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:31.228227 18680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:31.229154 18672 server_base.cc:1034] running on GCE node
I20260928 08:30:31.229758 18672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:31.230775 18672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:31.232048 18672 hybrid_clock.cc:648] HybridClock initialized: now 1790584231231964 us; error 54 us; skew 500 ppm
I20260928 08:30:31.232434 18672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:31 dist-test-slave-qp2n krb5kdc[18299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584231, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:31.235778 18672 init.cc:377] Logged in from keytab as kudu/127.15.134.195@KRBTEST.COM (short username kudu)
I20260928 08:30:31.237535 18672 webserver.cc:460] Webserver started at http://127.15.134.195:45329/ using document root <none> and password file <none>
I20260928 08:30:31.238250 18672 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:31.238351 18672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:31.238657 18672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:31.241753 18672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/data/instance:
uuid: "cc7a141f10dd4579bc4a6d4d64837378"
format_stamp: "Formatted at 2026-09-28 08:30:31 on dist-test-slave-qp2n"
server_key: "85913f522decce30864adfcb4e034a5a"
server_key_iv: "ef50ba48fa3b7c17f17d252a3e8c7fa3"
server_key_version: "encryptionkey@0"
I20260928 08:30:31.242537 18672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance:
uuid: "cc7a141f10dd4579bc4a6d4d64837378"
format_stamp: "Formatted at 2026-09-28 08:30:31 on dist-test-slave-qp2n"
server_key: "85913f522decce30864adfcb4e034a5a"
server_key_iv: "ef50ba48fa3b7c17f17d252a3e8c7fa3"
server_key_version: "encryptionkey@0"
I20260928 08:30:31.247829 18672 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:30:31.251520 18687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:31.253154 18672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 08:30:31.253337 18672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "cc7a141f10dd4579bc4a6d4d64837378"
format_stamp: "Formatted at 2026-09-28 08:30:31 on dist-test-slave-qp2n"
server_key: "85913f522decce30864adfcb4e034a5a"
server_key_iv: "ef50ba48fa3b7c17f17d252a3e8c7fa3"
server_key_version: "encryptionkey@0"
I20260928 08:30:31.253553 18672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:31.266845 18672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:31.271612 18672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:31.271922 18672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:31.272948 18672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:30:31.274302 18672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:31.274412 18672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:31.274561 18672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:31.274637 18672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:31.309577 18672 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:38321
I20260928 08:30:31.309625 18799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:38321 every 8 connection(s)
I20260928 08:30:31.310894 18672 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:30:31.319547 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 18672
I20260928 08:30:31.319761 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance
I20260928 08:30:31.320384 15899 external_mini_cluster.cc:1428] Setting key afbb157807c6e41aac60f5e164296070
Sep 28 08:30:31 dist-test-slave-qp2n krb5kdc[18299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584231, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:31.328410 18533 heartbeater.cc:503] Master 127.15.134.254:33189 was elected leader, sending a full tablet report...
I20260928 08:30:31.333174 18535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:31.313773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:57095 (local address 127.15.134.254:33189)
0928 08:30:31.313936 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:31.313942 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:31.315400 (+ 1458us) server_negotiation.cc:407] Connection header received
0928 08:30:31.317358 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:31.317363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:31.317533 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:31.317640 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:31.319388 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:31.320580 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:31.322844 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:31.323500 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:31.323643 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:31.327092 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:31.327097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:31.327100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:31.327132 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:31.330083 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:31.330898 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:31.330902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:31.330904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:31.331026 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:31.331528 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:31.331532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:31.331534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:31.331793 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:31.331914 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:31.332768 (+ 854us) server_negotiation.cc:299] Negotiation successful
0928 08:30:31.332933 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 08:30:31.334694 18802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:31.314262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:33189 (local address 127.15.134.195:57095)
0928 08:30:31.315218 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:31.315261 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:31.317021 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:31.317962 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:31.317972 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:31.318569 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:31.319207 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:31.319224 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:31.320760 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:31.320766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:31.322696 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:31.322710 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:31.323735 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:31.323744 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:31.323981 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:31.324731 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:31.324753 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:31.326857 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:31.330238 (+ 3381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:31.330247 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:31.330264 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:31.330795 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:31.331233 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:31.331237 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:31.331239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:31.331430 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:31.331984 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:31.331990 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:31.332377 (+ 387us) client_negotiation.cc:769] Sending connection context
0928 08:30:31.332743 (+ 366us) client_negotiation.cc:240] Negotiation successful
0928 08:30:31.333159 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":300,"threads_started":1}
I20260928 08:30:31.336103 18800 heartbeater.cc:347] Connected to a master server at 127.15.134.254:33189
I20260928 08:30:31.336472 18800 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:31.337244 18800 heartbeater.cc:511] Master 127.15.134.254:33189 requested a full tablet report, sending...
I20260928 08:30:31.338569 18344 ts_manager.cc:194] Registered new tserver with Master: cc7a141f10dd4579bc4a6d4d64837378 (127.15.134.195:38321)
I20260928 08:30:31.339464 18344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.195@KRBTEST.COM'} at 127.15.134.195:57095
I20260928 08:30:31.349121 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 08:30:31.381955 18535 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:30:31.379350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47238 (local address 127.15.134.254:33189)
0928 08:30:31.379542 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:31.379546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:31.380044 (+ 498us) server_negotiation.cc:407] Connection header received
0928 08:30:31.380178 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:31.380182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:31.380255 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:31.380376 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:31.381700 (+ 1324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47238: BlockingRecv error: recv got EOF from 127.0.0.1:47238 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 08:30:31.385919 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 18405
W20260928 08:30:31.390663 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 18405: Not authorized: Client connection negotiation failed: client connection to 127.15.134.193:40147: server requires authentication, but client does not have Kerberos credentials available
I20260928 08:30:31.390828 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 18405
W20260928 08:30:31.390892 18811 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:30:31.386913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45358 (local address 127.15.134.193:40147)
0928 08:30:31.389148 (+ 2235us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:31.389155 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:31.389182 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:30:31.389243 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:31.389249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:31.389500 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:31.389682 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:31.390523 (+ 841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45358: BlockingRecv error: recv got EOF from 127.0.0.1:45358 (error 108)
Metrics: {"server-negotiator.queue_time_us":2082,"thread_start_us":1864,"threads_started":1}
I20260928 08:30:31.407868 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 18539
W20260928 08:30:31.410665 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 18539: Not authorized: Client connection negotiation failed: client connection to 127.15.134.194:40827: server requires authentication, but client does not have Kerberos credentials available
I20260928 08:30:31.410846 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 18539
I20260928 08:30:31.427685 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 18672
W20260928 08:30:31.430749 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 18672: Not authorized: Client connection negotiation failed: client connection to 127.15.134.195:38321: server requires authentication, but client does not have Kerberos credentials available
I20260928 08:30:31.430873 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 18672
I20260928 08:30:31.448285 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 18312
I20260928 08:30:31.449695 18374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:31.555258 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 18312
2026-09-28T08:30:31Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3647 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:30:31 dist-test-slave-qp2n krb5kdc[18817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:30:31 dist-test-slave-qp2n krb5kdc[18817](info): set up 2 sockets
Sep 28 08:30:31 dist-test-slave-qp2n krb5kdc[18817](info): commencing operation
krb5kdc: starting...
W20260928 08:30:32.686643 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:30:32 dist-test-slave-qp2n krb5kdc[18817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:30:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:30:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:30:32.890206 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:44753
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:35621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:44753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:33.221295 18830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:33.221692 18830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:33.221859 18830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:33.229185 18830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:30:33.229342 18830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:33.229458 18830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:33.229561 18830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:30:33.229650 18830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:30:33.239674 18830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:35621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:44753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:44753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:33.241086 18830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:33.242431 18830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:33.254468 18835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:33.254546 18836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:33.254498 18838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:33.255622 18830 server_base.cc:1034] running on GCE node
I20260928 08:30:33.256397 18830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:33.257908 18830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:33.259169 18830 hybrid_clock.cc:648] HybridClock initialized: now 1790584233259099 us; error 54 us; skew 500 ppm
I20260928 08:30:33.259634 18830 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:33 dist-test-slave-qp2n krb5kdc[18817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584233, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:33.267706 18830 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:30:33.269258 18830 webserver.cc:460] Webserver started at http://127.15.134.254:42917/ using document root <none> and password file <none>
I20260928 08:30:33.269958 18830 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:33.270061 18830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:33.270313 18830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:33.273460 18830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "93ebc2ac735646d094996b119766a78f"
format_stamp: "Formatted at 2026-09-28 08:30:33 on dist-test-slave-qp2n"
server_key: "e64aaa53e1813a1ea5c2ae6a479d6735"
server_key_iv: "c9164cbad43f437f84cb10144889fe53"
server_key_version: "encryptionkey@0"
I20260928 08:30:33.274236 18830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "93ebc2ac735646d094996b119766a78f"
format_stamp: "Formatted at 2026-09-28 08:30:33 on dist-test-slave-qp2n"
server_key: "e64aaa53e1813a1ea5c2ae6a479d6735"
server_key_iv: "c9164cbad43f437f84cb10144889fe53"
server_key_version: "encryptionkey@0"
I20260928 08:30:33.279560 18830 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:30:33.283393 18845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:33.284906 18830 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:30:33.285100 18830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "93ebc2ac735646d094996b119766a78f"
format_stamp: "Formatted at 2026-09-28 08:30:33 on dist-test-slave-qp2n"
server_key: "e64aaa53e1813a1ea5c2ae6a479d6735"
server_key_iv: "c9164cbad43f437f84cb10144889fe53"
server_key_version: "encryptionkey@0"
I20260928 08:30:33.285305 18830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:33.314926 18830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:33.319343 18830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:33.319631 18830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:33.338919 18830 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:44753
I20260928 08:30:33.339131 18896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:44753 every 8 connection(s)
I20260928 08:30:33.340513 18830 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:30:33.342965 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 18830
I20260928 08:30:33.343200 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:30:33.343849 15899 external_mini_cluster.cc:1428] Setting key cc608079cbab10348fe884406db74d1f
I20260928 08:30:33.344906 18897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 08:30:33 dist-test-slave-qp2n krb5kdc[18817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:33.361336 18897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93ebc2ac735646d094996b119766a78f: Bootstrap starting.
I20260928 08:30:33.365513 18897 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93ebc2ac735646d094996b119766a78f: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:33.366955 18897 log.cc:824] T 00000000000000000000000000000000 P 93ebc2ac735646d094996b119766a78f: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:33.369613 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:33.345827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.15.134.254:44753)
0928 08:30:33.346652 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:33.346664 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:33.346705 (+ 41us) server_negotiation.cc:407] Connection header received
0928 08:30:33.348330 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:33.348354 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:33.348946 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:33.349442 (+ 496us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:33.351508 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:33.353084 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:33.355033 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:33.355759 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:33.355937 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:33.361254 (+ 5317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:33.361263 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:33.361281 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:33.361341 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:33.365019 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:33.365807 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:33.365817 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:33.365825 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:33.366085 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:33.366504 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:33.366512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:33.366516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:33.367303 (+ 787us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:33.367660 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:33.368095 (+ 435us) server_negotiation.cc:299] Negotiation successful
0928 08:30:33.368529 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":390,"threads_started":1}
I20260928 08:30:33.370389 18897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93ebc2ac735646d094996b119766a78f: No bootstrap required, opened a new log
I20260928 08:30:33.374264 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:30:33.378777 18897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93ebc2ac735646d094996b119766a78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ebc2ac735646d094996b119766a78f" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44753 } }
I20260928 08:30:33.379192 18897 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93ebc2ac735646d094996b119766a78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:33.379312 18897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93ebc2ac735646d094996b119766a78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93ebc2ac735646d094996b119766a78f, State: Initialized, Role: FOLLOWER
I20260928 08:30:33.379909 18897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93ebc2ac735646d094996b119766a78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ebc2ac735646d094996b119766a78f" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44753 } }
I20260928 08:30:33.380158 18897 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93ebc2ac735646d094996b119766a78f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:30:33.380306 18897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93ebc2ac735646d094996b119766a78f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:30:33.380489 18897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93ebc2ac735646d094996b119766a78f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:33.382771 18897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93ebc2ac735646d094996b119766a78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ebc2ac735646d094996b119766a78f" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44753 } }
I20260928 08:30:33.383291 18897 leader_election.cc:302] T 00000000000000000000000000000000 P 93ebc2ac735646d094996b119766a78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93ebc2ac735646d094996b119766a78f; no voters:
I20260928 08:30:33.383909 18897 leader_election.cc:288] T 00000000000000000000000000000000 P 93ebc2ac735646d094996b119766a78f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:30:33.384047 18903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93ebc2ac735646d094996b119766a78f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:33.385186 18903 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93ebc2ac735646d094996b119766a78f [term 1 LEADER]: Becoming Leader. State: Replica: 93ebc2ac735646d094996b119766a78f, State: Running, Role: LEADER
I20260928 08:30:33.385761 18903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93ebc2ac735646d094996b119766a78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ebc2ac735646d094996b119766a78f" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44753 } }
I20260928 08:30:33.386273 18897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93ebc2ac735646d094996b119766a78f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:30:33.388827 18905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93ebc2ac735646d094996b119766a78f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93ebc2ac735646d094996b119766a78f. Latest consensus state: current_term: 1 leader_uuid: "93ebc2ac735646d094996b119766a78f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ebc2ac735646d094996b119766a78f" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44753 } } }
I20260928 08:30:33.389292 18905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93ebc2ac735646d094996b119766a78f [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:33.389421 18904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93ebc2ac735646d094996b119766a78f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93ebc2ac735646d094996b119766a78f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93ebc2ac735646d094996b119766a78f" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44753 } } }
I20260928 08:30:33.389788 18904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93ebc2ac735646d094996b119766a78f [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:33.398881 18911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:30:33.404733 18911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:30:33.415565 18911 catalog_manager.cc:1347] Generated new cluster ID: 11b1a2f56115406a8ac0e1991739f057
I20260928 08:30:33.415699 18911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:30:33.426833 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:33.417738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35186 (local address 127.15.134.254:44753)
0928 08:30:33.417931 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:33.417938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:33.417965 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:30:33.418183 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:33.418190 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:33.418260 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:33.418323 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:33.419651 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:33.420943 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:33.422722 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:33.423443 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:33.423596 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:33.424801 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:33.424809 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:33.424814 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:30:33.425731 (+ 917us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:33.425790 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:30:33.425830 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:33.425954 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:33.426103 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 08:30:33.426215 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:30:33.432107 18911 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:30:33.433478 18911 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:30:33.440428 18911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93ebc2ac735646d094996b119766a78f: Generated new TSK 0
I20260928 08:30:33.441370 18911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:30:33.455752 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 18830
I20260928 08:30:33.457659 18892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:33.548098 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 18830
2026-09-28T08:30:33Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1988 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:30:33 dist-test-slave-qp2n krb5kdc[18931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:30:33 dist-test-slave-qp2n krb5kdc[18931](info): set up 2 sockets
Sep 28 08:30:33 dist-test-slave-qp2n krb5kdc[18931](info): commencing operation
krb5kdc: starting...
W20260928 08:30:34.658558 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:30:34 dist-test-slave-qp2n krb5kdc[18931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:30:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:30:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:30:34.862808 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:40457
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:40133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:40457
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:35.202127 18944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:35.202428 18944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:35.202589 18944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:35.209805 18944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:30:35.209911 18944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:35.210004 18944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:35.210111 18944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:30:35.210171 18944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:30:35.220171 18944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:40133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:40457
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:40457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:35.221546 18944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:35.222906 18944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:35.235725 18952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:35.235702 18950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:35.235684 18949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:35.238389 18944 server_base.cc:1034] running on GCE node
I20260928 08:30:35.239359 18944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:35.240906 18944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:35.242120 18944 hybrid_clock.cc:648] HybridClock initialized: now 1790584235242075 us; error 39 us; skew 500 ppm
I20260928 08:30:35.242478 18944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:35 dist-test-slave-qp2n krb5kdc[18931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584235, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:35.246086 18944 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:30:35.247933 18944 webserver.cc:460] Webserver started at http://127.15.134.254:40513/ using document root <none> and password file <none>
I20260928 08:30:35.248842 18944 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:35.248989 18944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:35.249389 18944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:35.252539 18944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "93210822847d48f49c4e30c6beaca230"
format_stamp: "Formatted at 2026-09-28 08:30:35 on dist-test-slave-qp2n"
server_key: "1fee6624abc1699c916eb33860cc2c67"
server_key_iv: "3ff2af899d9d763e8fa56d0c836186a6"
server_key_version: "encryptionkey@0"
I20260928 08:30:35.253463 18944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "93210822847d48f49c4e30c6beaca230"
format_stamp: "Formatted at 2026-09-28 08:30:35 on dist-test-slave-qp2n"
server_key: "1fee6624abc1699c916eb33860cc2c67"
server_key_iv: "3ff2af899d9d763e8fa56d0c836186a6"
server_key_version: "encryptionkey@0"
I20260928 08:30:35.259335 18944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260928 08:30:35.263119 18959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:35.264923 18944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:30:35.265123 18944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "93210822847d48f49c4e30c6beaca230"
format_stamp: "Formatted at 2026-09-28 08:30:35 on dist-test-slave-qp2n"
server_key: "1fee6624abc1699c916eb33860cc2c67"
server_key_iv: "3ff2af899d9d763e8fa56d0c836186a6"
server_key_version: "encryptionkey@0"
I20260928 08:30:35.265302 18944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:35.286316 18944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:35.290904 18944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:35.291199 18944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:35.310350 18944 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:40457
I20260928 08:30:35.310456 19010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:40457 every 8 connection(s)
I20260928 08:30:35.311712 18944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:30:35.313623 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 18944
I20260928 08:30:35.313853 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:30:35.314519 15899 external_mini_cluster.cc:1428] Setting key 35c44c0e81eb43b6bb4499124ae6064d
I20260928 08:30:35.316169 19011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 08:30:35 dist-test-slave-qp2n krb5kdc[18931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:35.330592 19011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93210822847d48f49c4e30c6beaca230: Bootstrap starting.
I20260928 08:30:35.334460 19011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93210822847d48f49c4e30c6beaca230: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:35.335624 19011 log.cc:824] T 00000000000000000000000000000000 P 93210822847d48f49c4e30c6beaca230: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:35.338769 19014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:35.316785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58806 (local address 127.15.134.254:40457)
0928 08:30:35.317583 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:35.317598 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:35.317647 (+ 49us) server_negotiation.cc:407] Connection header received
0928 08:30:35.319549 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:35.319575 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:35.320098 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:35.320806 (+ 708us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:35.322121 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:35.324296 (+ 2175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:35.325609 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:35.326651 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:35.326850 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:35.330919 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:35.330927 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:35.330941 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:35.330978 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:35.334136 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:35.334936 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:35.334943 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:35.334949 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:35.335115 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:35.335518 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:35.335522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:35.335524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:35.336091 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:35.336430 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:35.336912 (+ 482us) server_negotiation.cc:299] Negotiation successful
0928 08:30:35.337401 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":318,"threads_started":1}
I20260928 08:30:35.339444 19011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93210822847d48f49c4e30c6beaca230: No bootstrap required, opened a new log
I20260928 08:30:35.347504 19011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93210822847d48f49c4e30c6beaca230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93210822847d48f49c4e30c6beaca230" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 40457 } }
I20260928 08:30:35.347889 19011 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93210822847d48f49c4e30c6beaca230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:35.348006 19011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93210822847d48f49c4e30c6beaca230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93210822847d48f49c4e30c6beaca230, State: Initialized, Role: FOLLOWER
I20260928 08:30:35.348745 19011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93210822847d48f49c4e30c6beaca230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93210822847d48f49c4e30c6beaca230" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 40457 } }
I20260928 08:30:35.348975 19011 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93210822847d48f49c4e30c6beaca230 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:30:35.349130 19011 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93210822847d48f49c4e30c6beaca230 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:30:35.349303 19011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93210822847d48f49c4e30c6beaca230 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:35.351461 19011 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93210822847d48f49c4e30c6beaca230 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93210822847d48f49c4e30c6beaca230" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 40457 } }
I20260928 08:30:35.351969 19011 leader_election.cc:302] T 00000000000000000000000000000000 P 93210822847d48f49c4e30c6beaca230 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93210822847d48f49c4e30c6beaca230; no voters:
I20260928 08:30:35.352591 19011 leader_election.cc:288] T 00000000000000000000000000000000 P 93210822847d48f49c4e30c6beaca230 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:30:35.352859 19016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93210822847d48f49c4e30c6beaca230 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:35.353550 19016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93210822847d48f49c4e30c6beaca230 [term 1 LEADER]: Becoming Leader. State: Replica: 93210822847d48f49c4e30c6beaca230, State: Running, Role: LEADER
I20260928 08:30:35.354096 19016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93210822847d48f49c4e30c6beaca230 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93210822847d48f49c4e30c6beaca230" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 40457 } }
I20260928 08:30:35.354712 19011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93210822847d48f49c4e30c6beaca230 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:30:35.357141 19018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93210822847d48f49c4e30c6beaca230 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93210822847d48f49c4e30c6beaca230. Latest consensus state: current_term: 1 leader_uuid: "93210822847d48f49c4e30c6beaca230" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93210822847d48f49c4e30c6beaca230" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 40457 } } }
I20260928 08:30:35.357224 19017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93210822847d48f49c4e30c6beaca230 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93210822847d48f49c4e30c6beaca230" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93210822847d48f49c4e30c6beaca230" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 40457 } } }
I20260928 08:30:35.357544 19018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93210822847d48f49c4e30c6beaca230 [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:35.357635 19017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93210822847d48f49c4e30c6beaca230 [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:35.358471 19023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:30:35.364327 19023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:30:35.373256 19023 catalog_manager.cc:1347] Generated new cluster ID: 997d472f7ab940b1ab4611287ec9dd54
I20260928 08:30:35.373395 19023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:30:35.383073 19023 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:30:35.385094 19023 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:30:35.403681 19023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93210822847d48f49c4e30c6beaca230: Generated new TSK 0
I20260928 08:30:35.404889 19023 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.134.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
I20260928 08:30:35.452184 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:0
--local_ip_for_outbound_sockets=127.15.134.193
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:40457
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:40133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:35.791152 19037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:35.791484 19037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:35.791663 19037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:35.799201 19037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:35.799360 19037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:35.799543 19037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:30:35.812453 19037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:40133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:40457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:35.814105 19037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:35.815446 19037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:35.829368 19043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:35.829448 19045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:35.829397 19042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:35.830164 19037 server_base.cc:1034] running on GCE node
I20260928 08:30:35.830871 19037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:35.831880 19037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:35.833135 19037 hybrid_clock.cc:648] HybridClock initialized: now 1790584235833093 us; error 57 us; skew 500 ppm
I20260928 08:30:35.833501 19037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:35 dist-test-slave-qp2n krb5kdc[18931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584235, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:35.836872 19037 init.cc:377] Logged in from keytab as kudu/127.15.134.193@KRBTEST.COM (short username kudu)
I20260928 08:30:35.838605 19037 webserver.cc:460] Webserver started at http://127.15.134.193:36575/ using document root <none> and password file <none>
I20260928 08:30:35.839332 19037 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:35.839483 19037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:35.839799 19037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:35.842869 19037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/data/instance:
uuid: "2ac56192a491456b957c1929f3f35df3"
format_stamp: "Formatted at 2026-09-28 08:30:35 on dist-test-slave-qp2n"
server_key: "1344a1d9c96beb5db72c564b195f7241"
server_key_iv: "c47594f7fd3dbc284cde1d99e6190bc2"
server_key_version: "encryptionkey@0"
I20260928 08:30:35.843655 19037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance:
uuid: "2ac56192a491456b957c1929f3f35df3"
format_stamp: "Formatted at 2026-09-28 08:30:35 on dist-test-slave-qp2n"
server_key: "1344a1d9c96beb5db72c564b195f7241"
server_key_iv: "c47594f7fd3dbc284cde1d99e6190bc2"
server_key_version: "encryptionkey@0"
I20260928 08:30:35.848994 19037 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 08:30:35.852941 19052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:35.854592 19037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:30:35.854784 19037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "2ac56192a491456b957c1929f3f35df3"
format_stamp: "Formatted at 2026-09-28 08:30:35 on dist-test-slave-qp2n"
server_key: "1344a1d9c96beb5db72c564b195f7241"
server_key_iv: "c47594f7fd3dbc284cde1d99e6190bc2"
server_key_version: "encryptionkey@0"
I20260928 08:30:35.855034 19037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:35.877610 19037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:35.882409 19037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:35.882889 19037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:35.884076 19037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:30:35.885633 19037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:35.885754 19037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:35.885921 19037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:35.885993 19037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:35.920284 19037 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:42043
I20260928 08:30:35.920336 19164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:42043 every 8 connection(s)
I20260928 08:30:35.921693 19037 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:30:35.922730 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 19037
I20260928 08:30:35.922956 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance
I20260928 08:30:35.923734 15899 external_mini_cluster.cc:1428] Setting key 396e8bf3e341c1779d067c613375586b
Sep 28 08:30:35 dist-test-slave-qp2n krb5kdc[18931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584235, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.134.194@KRBTEST.COM; defaulting to no policy
I20260928 08:30:35.943593 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:35.924476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:58291 (local address 127.15.134.254:40457)
0928 08:30:35.925736 (+ 1260us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:35.925743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:35.926181 (+ 438us) server_negotiation.cc:407] Connection header received
0928 08:30:35.929042 (+ 2861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:35.929046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:35.929110 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:35.929203 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:35.931278 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:35.932000 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:35.933710 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:35.934112 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:35.934258 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:35.937749 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:35.937754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:35.937757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:35.937796 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:35.940421 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:35.941224 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:35.941229 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:35.941231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:35.941344 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:35.941789 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:35.941798 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:35.941800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:35.942057 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:35.942209 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:35.942988 (+ 779us) server_negotiation.cc:299] Negotiation successful
0928 08:30:35.943217 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1133,"thread_start_us":255,"threads_started":1}
I20260928 08:30:35.944586 19168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:35.925067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:40457 (local address 127.15.134.193:58291)
0928 08:30:35.925996 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:35.926053 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:35.928749 (+ 2696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:35.929485 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:35.929499 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:35.930268 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:35.931082 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:35.931107 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:35.932168 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:35.932175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:35.933554 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:35.933566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:35.934356 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:35.934365 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:35.934598 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:35.935285 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:35.935309 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:35.937523 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:35.940582 (+ 3059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:35.940591 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:35.940630 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:35.941083 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:35.941455 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:35.941461 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:35.941464 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:35.941656 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:35.942197 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:35.942205 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:35.942611 (+ 406us) client_negotiation.cc:769] Sending connection context
0928 08:30:35.942876 (+ 265us) client_negotiation.cc:240] Negotiation successful
0928 08:30:35.943464 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":360,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.194@KRBTEST.COM" created.
I20260928 08:30:35.946564 19165 heartbeater.cc:347] Connected to a master server at 127.15.134.254:40457
I20260928 08:30:35.946967 19165 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:35.947984 19165 heartbeater.cc:511] Master 127.15.134.254:40457 requested a full tablet report, sending...
I20260928 08:30:35.950562 18976 ts_manager.cc:194] Registered new tserver with Master: 2ac56192a491456b957c1929f3f35df3 (127.15.134.193:42043)
I20260928 08:30:35.952338 18976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.193@KRBTEST.COM'} at 127.15.134.193:58291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
I20260928 08:30:35.972158 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:0
--local_ip_for_outbound_sockets=127.15.134.194
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:40457
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:40133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:36.297314 19171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:36.297612 19171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:36.297794 19171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:36.304914 19171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:36.305032 19171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:36.305207 19171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:30:36.314817 19171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:40133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:40457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:36.316061 19171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:36.317428 19171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:36.330348 19177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:36.330437 19176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:36.330466 19179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:36.331182 19171 server_base.cc:1034] running on GCE node
I20260928 08:30:36.331758 19171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:36.332751 19171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:36.334010 19171 hybrid_clock.cc:648] HybridClock initialized: now 1790584236333950 us; error 43 us; skew 500 ppm
I20260928 08:30:36.334370 19171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:36 dist-test-slave-qp2n krb5kdc[18931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584236, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:36.337709 19171 init.cc:377] Logged in from keytab as kudu/127.15.134.194@KRBTEST.COM (short username kudu)
I20260928 08:30:36.339497 19171 webserver.cc:460] Webserver started at http://127.15.134.194:36751/ using document root <none> and password file <none>
I20260928 08:30:36.340229 19171 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:36.340376 19171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:36.340754 19171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:36.343677 19171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/data/instance:
uuid: "4daaeca2966c45f8ac998741530cbe5e"
format_stamp: "Formatted at 2026-09-28 08:30:36 on dist-test-slave-qp2n"
server_key: "6afdb42dad9df8eb0bc847b4af9725a4"
server_key_iv: "33c58437510957da0447e9e033755e1a"
server_key_version: "encryptionkey@0"
I20260928 08:30:36.344558 19171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance:
uuid: "4daaeca2966c45f8ac998741530cbe5e"
format_stamp: "Formatted at 2026-09-28 08:30:36 on dist-test-slave-qp2n"
server_key: "6afdb42dad9df8eb0bc847b4af9725a4"
server_key_iv: "33c58437510957da0447e9e033755e1a"
server_key_version: "encryptionkey@0"
I20260928 08:30:36.349926 19171 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 08:30:36.353953 19186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:36.355612 19171 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 08:30:36.355800 19171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "4daaeca2966c45f8ac998741530cbe5e"
format_stamp: "Formatted at 2026-09-28 08:30:36 on dist-test-slave-qp2n"
server_key: "6afdb42dad9df8eb0bc847b4af9725a4"
server_key_iv: "33c58437510957da0447e9e033755e1a"
server_key_version: "encryptionkey@0"
I20260928 08:30:36.356074 19171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:36.375095 19171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:36.379926 19171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:36.380257 19171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:36.381474 19171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:30:36.382990 19171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:36.383064 19171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:36.383222 19171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:36.383286 19171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:36.415824 19171 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:41917
I20260928 08:30:36.415912 19298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:41917 every 8 connection(s)
I20260928 08:30:36.417192 19171 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:30:36.424137 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 19171
I20260928 08:30:36.424329 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance
I20260928 08:30:36.425033 15899 external_mini_cluster.cc:1428] Setting key 40d79e0787b7d2c121e26d9e85bd0f8e
Sep 28 08:30:36 dist-test-slave-qp2n krb5kdc[18931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584236, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:36.440940 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:36.420783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:52373 (local address 127.15.134.254:40457)
0928 08:30:36.421109 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:36.421118 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:36.422396 (+ 1278us) server_negotiation.cc:407] Connection header received
0928 08:30:36.425433 (+ 3037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:36.425441 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:36.425603 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:36.425779 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:36.428087 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:36.428924 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:36.430530 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:36.430984 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:36.431136 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:36.435076 (+ 3940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:36.435081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:36.435085 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:36.435116 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:36.437778 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:36.438509 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:36.438514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:36.438518 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:36.438737 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:36.439178 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:36.439184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:36.439188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:36.439479 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:36.439647 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:36.440339 (+ 692us) server_negotiation.cc:299] Negotiation successful
0928 08:30:36.440567 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 08:30:36.442025 19301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:36.421202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:40457 (local address 127.15.134.194:52373)
0928 08:30:36.422216 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:36.422275 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:36.425052 (+ 2777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:36.426311 (+ 1259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:36.426325 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:36.427086 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:36.427885 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:36.427912 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:36.429079 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:36.429086 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:36.430365 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:36.430378 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:36.431230 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:36.431239 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:36.431508 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:36.432289 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:36.432319 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:36.434842 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:36.437924 (+ 3082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:36.437933 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:36.437950 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:36.438378 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:36.438861 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:36.438867 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:36.438871 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:36.439063 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:36.439607 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:36.439616 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:36.439966 (+ 350us) client_negotiation.cc:769] Sending connection context
0928 08:30:36.440240 (+ 274us) client_negotiation.cc:240] Negotiation successful
0928 08:30:36.440822 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":414,"threads_started":1}
WARNING: no policy specified for kudu/127.15.134.195@KRBTEST.COM; defaulting to no policy
I20260928 08:30:36.443935 19299 heartbeater.cc:347] Connected to a master server at 127.15.134.254:40457
I20260928 08:30:36.444238 19299 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:36.445161 19299 heartbeater.cc:511] Master 127.15.134.254:40457 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.195@KRBTEST.COM" created.
I20260928 08:30:36.446898 18976 ts_manager.cc:194] Registered new tserver with Master: 4daaeca2966c45f8ac998741530cbe5e (127.15.134.194:41917)
I20260928 08:30:36.447739 18976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.194@KRBTEST.COM'} at 127.15.134.194:52373
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
I20260928 08:30:36.471976 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:0
--local_ip_for_outbound_sockets=127.15.134.195
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:40457
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:40133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:36.818312 19304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:36.818588 19304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:36.818681 19304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:36.826061 19304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:36.826195 19304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:36.826400 19304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:30:36.837875 19304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:40133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:40457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:36.839211 19304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:36.840567 19304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:36.854159 19312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:36.854203 19309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:36.854095 19310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:36.857813 19304 server_base.cc:1034] running on GCE node
I20260928 08:30:36.858450 19304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:36.859546 19304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:36.860782 19304 hybrid_clock.cc:648] HybridClock initialized: now 1790584236860667 us; error 97 us; skew 500 ppm
I20260928 08:30:36.861176 19304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:36 dist-test-slave-qp2n krb5kdc[18931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584236, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:36.864519 19304 init.cc:377] Logged in from keytab as kudu/127.15.134.195@KRBTEST.COM (short username kudu)
I20260928 08:30:36.866087 19304 webserver.cc:460] Webserver started at http://127.15.134.195:38363/ using document root <none> and password file <none>
I20260928 08:30:36.866817 19304 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:36.866958 19304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:36.867269 19304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:36.870385 19304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/data/instance:
uuid: "620c272474ad482687b52e2309e02cf0"
format_stamp: "Formatted at 2026-09-28 08:30:36 on dist-test-slave-qp2n"
server_key: "d61159f95d91343d6ee74566e6253c88"
server_key_iv: "6ca901541c00c74b85944b53d41ee908"
server_key_version: "encryptionkey@0"
I20260928 08:30:36.871145 19304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance:
uuid: "620c272474ad482687b52e2309e02cf0"
format_stamp: "Formatted at 2026-09-28 08:30:36 on dist-test-slave-qp2n"
server_key: "d61159f95d91343d6ee74566e6253c88"
server_key_iv: "6ca901541c00c74b85944b53d41ee908"
server_key_version: "encryptionkey@0"
I20260928 08:30:36.876322 19304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 08:30:36.879895 19319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:36.881573 19304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:30:36.881775 19304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "620c272474ad482687b52e2309e02cf0"
format_stamp: "Formatted at 2026-09-28 08:30:36 on dist-test-slave-qp2n"
server_key: "d61159f95d91343d6ee74566e6253c88"
server_key_iv: "6ca901541c00c74b85944b53d41ee908"
server_key_version: "encryptionkey@0"
I20260928 08:30:36.882001 19304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:36.908476 19304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:36.913205 19304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:36.913538 19304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:36.914602 19304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:30:36.916016 19304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:36.916118 19304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:36.916268 19304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:36.916343 19304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:36.951402 19304 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:36219
I20260928 08:30:36.951457 19431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:36219 every 8 connection(s)
I20260928 08:30:36.952745 19304 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:30:36.953189 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 19304
I20260928 08:30:36.953415 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance
I20260928 08:30:36.954044 15899 external_mini_cluster.cc:1428] Setting key fc3b73d377bb1e1744cd6f4ccc0f16a2
I20260928 08:30:36.955890 19165 heartbeater.cc:503] Master 127.15.134.254:40457 was elected leader, sending a full tablet report...
Sep 28 08:30:36 dist-test-slave-qp2n krb5kdc[18931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584236, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:36.974268 19434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:36.955758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:44057 (local address 127.15.134.254:40457)
0928 08:30:36.960361 (+ 4603us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:36.960367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:36.960389 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:30:36.960430 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:36.960433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:36.960494 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:36.960577 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:36.962302 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:36.963098 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:36.964571 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:36.965059 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:36.965157 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:36.968492 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:36.968497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:36.968502 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:36.968537 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:36.971563 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:36.972204 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:36.972208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:36.972210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:36.972332 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:36.972784 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:36.972787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:36.972789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:36.973040 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:36.973148 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:36.973855 (+ 707us) server_negotiation.cc:299] Negotiation successful
0928 08:30:36.974009 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4439,"thread_start_us":4370,"threads_started":1}
I20260928 08:30:36.975201 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:36.956548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:40457 (local address 127.15.134.195:44057)
0928 08:30:36.957698 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:36.957748 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:36.959786 (+ 2038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:36.960928 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:36.960937 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:36.961560 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:36.962108 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:36.962125 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:36.963233 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:36.963238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:36.964450 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:36.964462 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:36.965245 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:36.965249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:36.965434 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:36.966132 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:36.966154 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:36.968251 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:36.971685 (+ 3434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:36.971691 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:36.971703 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:36.972104 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:36.972429 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:36.972432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:36.972434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:36.972599 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:36.973152 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:36.973158 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:36.973473 (+ 315us) client_negotiation.cc:769] Sending connection context
0928 08:30:36.973689 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 08:30:36.974108 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"thread_start_us":1071,"threads_started":1}
I20260928 08:30:36.976985 19432 heartbeater.cc:347] Connected to a master server at 127.15.134.254:40457
I20260928 08:30:36.977247 19432 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:36.977917 19432 heartbeater.cc:511] Master 127.15.134.254:40457 requested a full tablet report, sending...
I20260928 08:30:36.979279 18975 ts_manager.cc:194] Registered new tserver with Master: 620c272474ad482687b52e2309e02cf0 (127.15.134.195:36219)
I20260928 08:30:36.980130 18975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.195@KRBTEST.COM'} at 127.15.134.195:44057
I20260928 08:30:36.982416 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 08:30:37 dist-test-slave-qp2n krb5kdc[18931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584237, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 08:30:37 dist-test-slave-qp2n krb5kdc[18931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584237, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:37.031968 19434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:37.016917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58810 (local address 127.15.134.254:40457)
0928 08:30:37.017178 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:37.017186 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:37.017518 (+ 332us) server_negotiation.cc:407] Connection header received
0928 08:30:37.017671 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:37.017675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:37.017750 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:37.017854 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:37.019242 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:37.019977 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:37.021401 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:37.021862 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:37.021971 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:37.026325 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:37.026330 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:37.026333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:37.026375 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:37.029255 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:37.029976 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:37.029980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:37.029983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:37.030129 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:37.030564 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:37.030567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:37.030569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:37.030930 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:37.031108 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:37.031454 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 08:30:37.031623 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260928 08:30:37.032572 18975 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58810
I20260928 08:30:37.034489 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 19037
Sep 28 08:30:37 dist-test-slave-qp2n krb5kdc[18931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584237, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.15.134.193@KRBTEST.COM
I20260928 08:30:37.051692 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:37.035895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58518 (local address 127.15.134.193:42043)
0928 08:30:37.036528 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:37.036535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:37.036561 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:30:37.036700 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:37.036708 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:37.036963 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:37.037113 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:37.038394 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:37.039443 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:37.040780 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:37.041370 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:37.041472 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:37.045799 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:37.045814 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:37.045832 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:37.045871 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:37.048992 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:37.049687 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:37.049694 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:37.049696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:37.049916 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:37.050309 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:37.050312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:37.050314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:37.050661 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:37.050958 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:37.051193 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 08:30:37.051400 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":316,"threads_started":1}
W20260928 08:30:37.053053 19160 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58518
W20260928 08:30:37.054169 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 19037: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:30:37.054495 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 19037
I20260928 08:30:37.071867 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 19171
Sep 28 08:30:37 dist-test-slave-qp2n krb5kdc[18931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584237, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.15.134.194@KRBTEST.COM
I20260928 08:30:37.089043 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:37.072938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53284 (local address 127.15.134.194:41917)
0928 08:30:37.073533 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:37.073541 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:37.073567 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:30:37.073635 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:37.073642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:37.073857 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:37.073994 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:37.075262 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:37.076299 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:37.077780 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:37.078351 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:37.078453 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:37.082570 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:37.082581 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:37.082597 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:37.082637 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:37.086121 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:37.087065 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:37.087073 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:37.087075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:37.087207 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:37.087574 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:37.087577 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:37.087579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:37.087944 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:37.088170 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:37.088479 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 08:30:37.088746 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":319,"threads_started":1}
W20260928 08:30:37.090328 19294 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53284
W20260928 08:30:37.091462 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 19171: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:30:37.091615 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 19171
I20260928 08:30:37.107865 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 19304
Sep 28 08:30:37 dist-test-slave-qp2n krb5kdc[18931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584237, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.15.134.195@KRBTEST.COM
I20260928 08:30:37.124074 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:37.108844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34192 (local address 127.15.134.195:36219)
0928 08:30:37.109480 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:37.109487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:37.109515 (+ 28us) server_negotiation.cc:407] Connection header received
0928 08:30:37.109573 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:37.109579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:37.109800 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:37.109942 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:37.111151 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:37.112173 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:37.113482 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:37.114043 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:37.114140 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:37.118582 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:37.118603 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:37.118618 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:37.118657 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:37.121504 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:37.122134 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:37.122141 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:37.122143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:37.122293 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:37.122662 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:37.122666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:37.122668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:37.123039 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:37.123300 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:37.123538 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 08:30:37.123730 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":319,"threads_started":1}
W20260928 08:30:37.125484 19427 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34192
W20260928 08:30:37.126601 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 19304: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:30:37.126775 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 19304
I20260928 08:30:37.143170 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 18944
I20260928 08:30:37.144366 19006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:37.248831 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 18944
2026-09-28T08:30:37Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3705 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:30:37 dist-test-slave-qp2n krb5kdc[19450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:30:37 dist-test-slave-qp2n krb5kdc[19450](info): set up 2 sockets
Sep 28 08:30:37 dist-test-slave-qp2n krb5kdc[19450](info): commencing operation
krb5kdc: starting...
W20260928 08:30:38.362880 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:30:38 dist-test-slave-qp2n krb5kdc[19450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:30:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:30:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:30:38.576875 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:46477
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:46477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:38.909780 19463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:38.910077 19463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:38.910284 19463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:38.917673 19463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:30:38.917790 19463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:38.917878 19463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:38.917970 19463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:30:38.918035 19463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:30:38.928315 19463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:46477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:46477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:38.929750 19463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:38.931111 19463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:38.943936 19468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:38.944008 19471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:38.943878 19469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:38.944977 19463 server_base.cc:1034] running on GCE node
I20260928 08:30:38.945964 19463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:38.947693 19463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:38.948920 19463 hybrid_clock.cc:648] HybridClock initialized: now 1790584238948854 us; error 56 us; skew 500 ppm
I20260928 08:30:38.949285 19463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:38 dist-test-slave-qp2n krb5kdc[19450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584238, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:38.958040 19463 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:30:38.959805 19463 webserver.cc:460] Webserver started at http://127.15.134.254:41447/ using document root <none> and password file <none>
I20260928 08:30:38.960588 19463 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:38.960807 19463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:38.961158 19463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:38.964273 19463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "8a905ffbd2124cd3b4c1fabebd977aec"
format_stamp: "Formatted at 2026-09-28 08:30:38 on dist-test-slave-qp2n"
server_key: "bfc59998a5fa3b2e6bcadcdedd6af9b9"
server_key_iv: "b9a60db341c1cef5baeb365554df273f"
server_key_version: "encryptionkey@0"
I20260928 08:30:38.965116 19463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "8a905ffbd2124cd3b4c1fabebd977aec"
format_stamp: "Formatted at 2026-09-28 08:30:38 on dist-test-slave-qp2n"
server_key: "bfc59998a5fa3b2e6bcadcdedd6af9b9"
server_key_iv: "b9a60db341c1cef5baeb365554df273f"
server_key_version: "encryptionkey@0"
I20260928 08:30:38.970577 19463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:30:38.974582 19478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:38.976172 19463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:30:38.976394 19463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "8a905ffbd2124cd3b4c1fabebd977aec"
format_stamp: "Formatted at 2026-09-28 08:30:38 on dist-test-slave-qp2n"
server_key: "bfc59998a5fa3b2e6bcadcdedd6af9b9"
server_key_iv: "b9a60db341c1cef5baeb365554df273f"
server_key_version: "encryptionkey@0"
I20260928 08:30:38.976719 19463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:38.999370 19463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:39.004038 19463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:39.004295 19463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:39.024331 19463 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:46477
I20260928 08:30:39.024350 19529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:46477 every 8 connection(s)
I20260928 08:30:39.025867 19463 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:30:39.027930 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 19463
I20260928 08:30:39.028184 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:30:39.028913 15899 external_mini_cluster.cc:1428] Setting key 95efb3b28fd0110441e0f6f4f740d393
I20260928 08:30:39.030665 19530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 08:30:39 dist-test-slave-qp2n krb5kdc[19450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:39.045527 19530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a905ffbd2124cd3b4c1fabebd977aec: Bootstrap starting.
I20260928 08:30:39.049302 19530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a905ffbd2124cd3b4c1fabebd977aec: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:39.050644 19530 log.cc:824] T 00000000000000000000000000000000 P 8a905ffbd2124cd3b4c1fabebd977aec: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:39.053318 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:39.030929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41948 (local address 127.15.134.254:46477)
0928 08:30:39.031717 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:39.031728 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:39.031770 (+ 42us) server_negotiation.cc:407] Connection header received
0928 08:30:39.033520 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:39.033541 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:39.033938 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:39.034294 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:39.036169 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:39.037685 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:39.039520 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:39.040223 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:39.040360 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:39.045488 (+ 5128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:39.045497 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:39.045518 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:39.045563 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:39.048887 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:39.049821 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:39.049828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:39.049835 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:39.050001 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:39.050435 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:39.050439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:39.050441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:39.051071 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:39.051476 (+ 405us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:39.051847 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 08:30:39.052263 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":360,"threads_started":1}
I20260928 08:30:39.053898 19530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a905ffbd2124cd3b4c1fabebd977aec: No bootstrap required, opened a new log
I20260928 08:30:39.061638 19530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a905ffbd2124cd3b4c1fabebd977aec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a905ffbd2124cd3b4c1fabebd977aec" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46477 } }
I20260928 08:30:39.062158 19530 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a905ffbd2124cd3b4c1fabebd977aec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:39.062289 19530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a905ffbd2124cd3b4c1fabebd977aec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a905ffbd2124cd3b4c1fabebd977aec, State: Initialized, Role: FOLLOWER
I20260928 08:30:39.063022 19530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a905ffbd2124cd3b4c1fabebd977aec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a905ffbd2124cd3b4c1fabebd977aec" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46477 } }
I20260928 08:30:39.063261 19530 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a905ffbd2124cd3b4c1fabebd977aec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:30:39.063364 19530 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a905ffbd2124cd3b4c1fabebd977aec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:30:39.063491 19530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a905ffbd2124cd3b4c1fabebd977aec [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:39.065757 19530 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a905ffbd2124cd3b4c1fabebd977aec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a905ffbd2124cd3b4c1fabebd977aec" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46477 } }
I20260928 08:30:39.066323 19530 leader_election.cc:302] T 00000000000000000000000000000000 P 8a905ffbd2124cd3b4c1fabebd977aec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a905ffbd2124cd3b4c1fabebd977aec; no voters:
I20260928 08:30:39.066869 19530 leader_election.cc:288] T 00000000000000000000000000000000 P 8a905ffbd2124cd3b4c1fabebd977aec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:30:39.067039 19535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a905ffbd2124cd3b4c1fabebd977aec [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:39.068311 19535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a905ffbd2124cd3b4c1fabebd977aec [term 1 LEADER]: Becoming Leader. State: Replica: 8a905ffbd2124cd3b4c1fabebd977aec, State: Running, Role: LEADER
I20260928 08:30:39.068943 19535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a905ffbd2124cd3b4c1fabebd977aec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a905ffbd2124cd3b4c1fabebd977aec" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46477 } }
I20260928 08:30:39.069118 19530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a905ffbd2124cd3b4c1fabebd977aec [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:30:39.072165 19536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a905ffbd2124cd3b4c1fabebd977aec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a905ffbd2124cd3b4c1fabebd977aec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a905ffbd2124cd3b4c1fabebd977aec" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46477 } } }
I20260928 08:30:39.072229 19537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a905ffbd2124cd3b4c1fabebd977aec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a905ffbd2124cd3b4c1fabebd977aec. Latest consensus state: current_term: 1 leader_uuid: "8a905ffbd2124cd3b4c1fabebd977aec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a905ffbd2124cd3b4c1fabebd977aec" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46477 } } }
I20260928 08:30:39.072556 19536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a905ffbd2124cd3b4c1fabebd977aec [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:39.072665 19537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a905ffbd2124cd3b4c1fabebd977aec [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:39.078296 19544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:30:39.085196 19544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:30:39.094334 19544 catalog_manager.cc:1347] Generated new cluster ID: ee3397bf39a2433c8bca9e0dd5b46993
I20260928 08:30:39.094512 19544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:30:39.118566 19544 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:30:39.120667 19544 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:30:39.132092 19544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a905ffbd2124cd3b4c1fabebd977aec: Generated new TSK 0
I20260928 08:30:39.133157 19544 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.134.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
I20260928 08:30:39.181583 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:0
--local_ip_for_outbound_sockets=127.15.134.193
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:46477
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:39.526225 19556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:39.526520 19556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:39.526623 19556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:39.534142 19556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:39.534266 19556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:39.534391 19556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:30:39.544404 19556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:46477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:39.545814 19556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:39.547175 19556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:39.561347 19561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:39.561458 19562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:39.561534 19564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:39.562728 19556 server_base.cc:1034] running on GCE node
I20260928 08:30:39.563385 19556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:39.564441 19556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:39.565686 19556 hybrid_clock.cc:648] HybridClock initialized: now 1790584239565645 us; error 66 us; skew 500 ppm
I20260928 08:30:39.565992 19556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:39 dist-test-slave-qp2n krb5kdc[19450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584239, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:39.569355 19556 init.cc:377] Logged in from keytab as kudu/127.15.134.193@KRBTEST.COM (short username kudu)
I20260928 08:30:39.571154 19556 webserver.cc:460] Webserver started at http://127.15.134.193:33419/ using document root <none> and password file <none>
I20260928 08:30:39.571866 19556 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:39.572010 19556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:39.572341 19556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:39.575289 19556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-0/data/instance:
uuid: "08fff7983a454c3181ccd315c45638f6"
format_stamp: "Formatted at 2026-09-28 08:30:39 on dist-test-slave-qp2n"
server_key: "176ba5b3d59d019e19059e97637f2e1e"
server_key_iv: "57e39b6c7ea7663b4d3968de6cfacaec"
server_key_version: "encryptionkey@0"
I20260928 08:30:39.576112 19556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance:
uuid: "08fff7983a454c3181ccd315c45638f6"
format_stamp: "Formatted at 2026-09-28 08:30:39 on dist-test-slave-qp2n"
server_key: "176ba5b3d59d019e19059e97637f2e1e"
server_key_iv: "57e39b6c7ea7663b4d3968de6cfacaec"
server_key_version: "encryptionkey@0"
I20260928 08:30:39.581897 19556 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 08:30:39.585590 19571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:39.587347 19556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:30:39.587558 19556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "08fff7983a454c3181ccd315c45638f6"
format_stamp: "Formatted at 2026-09-28 08:30:39 on dist-test-slave-qp2n"
server_key: "176ba5b3d59d019e19059e97637f2e1e"
server_key_iv: "57e39b6c7ea7663b4d3968de6cfacaec"
server_key_version: "encryptionkey@0"
I20260928 08:30:39.587823 19556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:39.607015 19556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:39.611793 19556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:39.612116 19556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:39.613179 19556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:30:39.614627 19556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:39.614753 19556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:39.614938 19556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:39.615021 19556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:39.651918 19556 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:41895
I20260928 08:30:39.651965 19683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:41895 every 8 connection(s)
I20260928 08:30:39.653493 19556 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:30:39.661789 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 19556
I20260928 08:30:39.662087 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance
I20260928 08:30:39.662784 15899 external_mini_cluster.cc:1428] Setting key 3d418f99ffb72bb4332fb4bd49550434
Sep 28 08:30:39 dist-test-slave-qp2n krb5kdc[19450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584239, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:39.675757 19686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:39.656406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:48575 (local address 127.15.134.254:46477)
0928 08:30:39.656896 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:39.656901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:39.657947 (+ 1046us) server_negotiation.cc:407] Connection header received
0928 08:30:39.659877 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:39.659883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:39.659979 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:39.660100 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:39.661809 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:39.662656 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:39.664792 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:39.665190 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:39.665329 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:39.669116 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:39.669122 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:39.669125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:39.669159 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:39.671951 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:39.672978 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:39.672984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:39.672986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:39.673171 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:39.673775 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:39.673779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:39.673782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:39.674094 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:39.674260 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:39.675265 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0928 08:30:39.675478 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":270,"threads_started":1}
I20260928 08:30:39.677450 19687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:39.657012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46477 (local address 127.15.134.193:48575)
0928 08:30:39.657760 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:39.657817 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:39.659581 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:39.660363 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:39.660372 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:39.660973 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:39.661613 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:39.661632 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:39.662825 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:39.662835 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:39.664532 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:39.664548 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:39.665426 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:39.665433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:39.665678 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:39.666451 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:39.666475 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:39.668849 (+ 2374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:39.672109 (+ 3260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:39.672121 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:39.672145 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:39.672824 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:39.673312 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:39.673320 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:39.673323 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:39.673619 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:39.674239 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:39.674250 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:39.674774 (+ 524us) client_negotiation.cc:769] Sending connection context
0928 08:30:39.675105 (+ 331us) client_negotiation.cc:240] Negotiation successful
0928 08:30:39.675745 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":317,"threads_started":1}
I20260928 08:30:39.679435 19684 heartbeater.cc:347] Connected to a master server at 127.15.134.254:46477
I20260928 08:30:39.679807 19684 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:39.680830 19684 heartbeater.cc:511] Master 127.15.134.254:46477 requested a full tablet report, sending...
I20260928 08:30:39.683338 19495 ts_manager.cc:194] Registered new tserver with Master: 08fff7983a454c3181ccd315c45638f6 (127.15.134.193:41895)
WARNING: no policy specified for kudu/127.15.134.194@KRBTEST.COM; defaulting to no policy
I20260928 08:30:39.685082 19495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.193@KRBTEST.COM'} at 127.15.134.193:48575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
I20260928 08:30:39.713137 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:0
--local_ip_for_outbound_sockets=127.15.134.194
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:46477
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:40.043150 19690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:40.043449 19690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:40.043623 19690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:40.051118 19690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:40.051247 19690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:40.051442 19690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:30:40.061764 19690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:46477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:40.063131 19690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:40.064503 19690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:40.078092 19695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:40.078286 19696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:40.078125 19698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:40.080092 19690 server_base.cc:1034] running on GCE node
I20260928 08:30:40.080920 19690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:40.082026 19690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:40.083240 19690 hybrid_clock.cc:648] HybridClock initialized: now 1790584240083174 us; error 57 us; skew 500 ppm
I20260928 08:30:40.083577 19690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:40 dist-test-slave-qp2n krb5kdc[19450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584240, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:40.087085 19690 init.cc:377] Logged in from keytab as kudu/127.15.134.194@KRBTEST.COM (short username kudu)
I20260928 08:30:40.088761 19690 webserver.cc:460] Webserver started at http://127.15.134.194:33689/ using document root <none> and password file <none>
I20260928 08:30:40.089484 19690 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:40.089612 19690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:40.089936 19690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:40.093171 19690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-1/data/instance:
uuid: "4aea2c3f8f204a29b27f925a681cdb1c"
format_stamp: "Formatted at 2026-09-28 08:30:40 on dist-test-slave-qp2n"
server_key: "880cede0f9e57de9305b8ef683161c92"
server_key_iv: "2502e1a4beb28b1b0659893fe139cf8b"
server_key_version: "encryptionkey@0"
I20260928 08:30:40.094127 19690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance:
uuid: "4aea2c3f8f204a29b27f925a681cdb1c"
format_stamp: "Formatted at 2026-09-28 08:30:40 on dist-test-slave-qp2n"
server_key: "880cede0f9e57de9305b8ef683161c92"
server_key_iv: "2502e1a4beb28b1b0659893fe139cf8b"
server_key_version: "encryptionkey@0"
I20260928 08:30:40.099577 19690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 08:30:40.103628 19705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:40.105038 19690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 08:30:40.105222 19690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "4aea2c3f8f204a29b27f925a681cdb1c"
format_stamp: "Formatted at 2026-09-28 08:30:40 on dist-test-slave-qp2n"
server_key: "880cede0f9e57de9305b8ef683161c92"
server_key_iv: "2502e1a4beb28b1b0659893fe139cf8b"
server_key_version: "encryptionkey@0"
I20260928 08:30:40.105420 19690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:40.126173 19690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:40.130223 19690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:40.130509 19690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:40.131476 19690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:30:40.133023 19690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:40.133096 19690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:40.133226 19690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:40.133275 19690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:40.168017 19690 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:34447
I20260928 08:30:40.168066 19817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:34447 every 8 connection(s)
I20260928 08:30:40.169448 19690 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:30:40.173413 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 19690
I20260928 08:30:40.173612 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance
I20260928 08:30:40.174125 15899 external_mini_cluster.cc:1428] Setting key a226c7cad3cf57c31a71a4dca93c36b8
Sep 28 08:30:40 dist-test-slave-qp2n krb5kdc[19450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584240, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:40.192111 19686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:40.172168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:43405 (local address 127.15.134.254:46477)
0928 08:30:40.172380 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:40.172387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:40.173876 (+ 1489us) server_negotiation.cc:407] Connection header received
0928 08:30:40.175914 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:40.175920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:40.176029 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:40.176151 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:40.177899 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:40.178708 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:40.180437 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:40.181300 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:40.181472 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:40.185787 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:40.185793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:40.185796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:40.185829 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:40.188496 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:40.189554 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:40.189558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:40.189561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:40.189689 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:40.190218 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:40.190222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:40.190224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:40.190475 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:40.190583 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:40.191660 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0928 08:30:40.191859 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 08:30:40.193766 19820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:40.172766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46477 (local address 127.15.134.194:43405)
0928 08:30:40.173664 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:40.173738 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:40.175593 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:40.176422 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:40.176431 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:40.177037 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:40.177705 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:40.177736 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:40.178891 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:40.178897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:40.180257 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:40.180271 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:40.181589 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:40.181598 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:40.182053 (+ 455us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:40.182920 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:40.182954 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:40.185520 (+ 2566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:40.188681 (+ 3161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:40.188693 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:40.188713 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:40.189410 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:40.189821 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:40.189828 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:40.189832 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:40.190096 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:40.190610 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:40.190621 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:40.191136 (+ 515us) client_negotiation.cc:769] Sending connection context
0928 08:30:40.191497 (+ 361us) client_negotiation.cc:240] Negotiation successful
0928 08:30:40.192104 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":361,"threads_started":1}
WARNING: no policy specified for kudu/127.15.134.195@KRBTEST.COM; defaulting to no policy
I20260928 08:30:40.195436 19818 heartbeater.cc:347] Connected to a master server at 127.15.134.254:46477
I20260928 08:30:40.195793 19818 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:40.196712 19818 heartbeater.cc:511] Master 127.15.134.254:46477 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.195@KRBTEST.COM" created.
I20260928 08:30:40.198482 19495 ts_manager.cc:194] Registered new tserver with Master: 4aea2c3f8f204a29b27f925a681cdb1c (127.15.134.194:34447)
I20260928 08:30:40.199452 19495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.194@KRBTEST.COM'} at 127.15.134.194:43405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
I20260928 08:30:40.224170 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:0
--local_ip_for_outbound_sockets=127.15.134.195
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:46477
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:40.571482 19823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:40.571834 19823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:40.572005 19823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:40.579613 19823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:40.579771 19823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:40.579986 19823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:30:40.590147 19823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:46477
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:40.591562 19823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:40.593067 19823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:40.607069 19829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:40.607144 19828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:40.607110 19831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:40.607784 19823 server_base.cc:1034] running on GCE node
I20260928 08:30:40.608834 19823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:40.609992 19823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:40.611320 19823 hybrid_clock.cc:648] HybridClock initialized: now 1790584240611230 us; error 108 us; skew 500 ppm
I20260928 08:30:40.611740 19823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:40 dist-test-slave-qp2n krb5kdc[19450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584240, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:40.615119 19823 init.cc:377] Logged in from keytab as kudu/127.15.134.195@KRBTEST.COM (short username kudu)
I20260928 08:30:40.616956 19823 webserver.cc:460] Webserver started at http://127.15.134.195:33097/ using document root <none> and password file <none>
I20260928 08:30:40.617772 19823 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:40.617877 19823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:40.618136 19823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:40.621194 19823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-2/data/instance:
uuid: "e65231bd77cd4f17b151280d24fecafc"
format_stamp: "Formatted at 2026-09-28 08:30:40 on dist-test-slave-qp2n"
server_key: "19dd7f7234d1f65d873fce2d24ee9abf"
server_key_iv: "ef06b65e968f5cf03a4158be26ecd914"
server_key_version: "encryptionkey@0"
I20260928 08:30:40.621960 19823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance:
uuid: "e65231bd77cd4f17b151280d24fecafc"
format_stamp: "Formatted at 2026-09-28 08:30:40 on dist-test-slave-qp2n"
server_key: "19dd7f7234d1f65d873fce2d24ee9abf"
server_key_iv: "ef06b65e968f5cf03a4158be26ecd914"
server_key_version: "encryptionkey@0"
I20260928 08:30:40.627295 19823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:30:40.630996 19838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:40.632452 19823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:30:40.632730 19823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "e65231bd77cd4f17b151280d24fecafc"
format_stamp: "Formatted at 2026-09-28 08:30:40 on dist-test-slave-qp2n"
server_key: "19dd7f7234d1f65d873fce2d24ee9abf"
server_key_iv: "ef06b65e968f5cf03a4158be26ecd914"
server_key_version: "encryptionkey@0"
I20260928 08:30:40.632947 19823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:40.652118 19823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:40.657644 19823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:40.657948 19823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:40.658959 19823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:30:40.660301 19823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:40.660377 19823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:40.660465 19823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:40.660508 19823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:40.688838 19684 heartbeater.cc:503] Master 127.15.134.254:46477 was elected leader, sending a full tablet report...
I20260928 08:30:40.696231 19823 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:38007
I20260928 08:30:40.696275 19950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:38007 every 8 connection(s)
I20260928 08:30:40.697577 19823 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:30:40.705921 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 19823
I20260928 08:30:40.706138 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance
I20260928 08:30:40.706693 15899 external_mini_cluster.cc:1428] Setting key 33f755581efbdc77ad15e4070ec4b095
Sep 28 08:30:40 dist-test-slave-qp2n krb5kdc[19450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584240, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:40.718945 19953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:40.700098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:39193 (local address 127.15.134.254:46477)
0928 08:30:40.700525 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:40.700529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:40.701555 (+ 1026us) server_negotiation.cc:407] Connection header received
0928 08:30:40.703729 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:40.703735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:40.703823 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:40.703897 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:40.706099 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:40.707295 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:40.709253 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:40.709836 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:40.709947 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:40.713383 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:40.713388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:40.713391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:40.713422 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:40.715886 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:40.716592 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:40.716596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:40.716599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:40.716794 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:40.717255 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:40.717259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:40.717262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:40.717548 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:40.717665 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:40.718437 (+ 772us) server_negotiation.cc:299] Negotiation successful
0928 08:30:40.718627 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":210,"threads_started":1}
I20260928 08:30:40.719707 19954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:40.700720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46477 (local address 127.15.134.195:39193)
0928 08:30:40.701401 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:40.701437 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:40.703416 (+ 1979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:40.704390 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:40.704401 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:40.705104 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:40.705890 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:40.705915 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:40.707449 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:40.707457 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:40.709083 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:40.709100 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:40.710068 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:40.710074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:40.710356 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:40.710970 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:40.710997 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:40.713189 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:40.716021 (+ 2832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:40.716028 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:40.716040 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:40.716481 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:40.716901 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:40.716904 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:40.716907 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:40.717100 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:40.717664 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:40.717672 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:40.717987 (+ 315us) client_negotiation.cc:769] Sending connection context
0928 08:30:40.718252 (+ 265us) client_negotiation.cc:240] Negotiation successful
0928 08:30:40.718623 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":252,"threads_started":1}
I20260928 08:30:40.721490 19951 heartbeater.cc:347] Connected to a master server at 127.15.134.254:46477
I20260928 08:30:40.721871 19951 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:40.722559 19951 heartbeater.cc:511] Master 127.15.134.254:46477 requested a full tablet report, sending...
I20260928 08:30:40.724015 19494 ts_manager.cc:194] Registered new tserver with Master: e65231bd77cd4f17b151280d24fecafc (127.15.134.195:38007)
I20260928 08:30:40.725003 19494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.195@KRBTEST.COM'} at 127.15.134.195:39193
I20260928 08:30:40.734983 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 08:30:40 dist-test-slave-qp2n krb5kdc[19450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 08:30:40 dist-test-slave-qp2n krb5kdc[19450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.193@KRBTEST.COM
I20260928 08:30:40.788023 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:40.770482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47724 (local address 127.15.134.193:41895)
0928 08:30:40.771248 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:40.771256 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:40.771286 (+ 30us) server_negotiation.cc:407] Connection header received
0928 08:30:40.771389 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:40.771396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:40.771629 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:40.771771 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:40.773351 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:40.774440 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:40.775801 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:40.776389 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:40.776494 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:40.782116 (+ 5622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:40.782133 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:40.782150 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:40.782201 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:40.785130 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:40.785834 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:40.785842 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:40.785844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:40.785974 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:40.786440 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:40.786443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:40.786445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:40.786826 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:40.787049 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:40.787361 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 08:30:40.787561 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":449,"threads_started":1}
Sep 28 08:30:40 dist-test-slave-qp2n krb5kdc[19450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:40.807874 19953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:40.793071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41954 (local address 127.15.134.254:46477)
0928 08:30:40.793343 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:40.793350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:40.793554 (+ 204us) server_negotiation.cc:407] Connection header received
0928 08:30:40.793782 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:40.793787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:40.793866 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:40.793970 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:40.795413 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:40.796136 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:40.797455 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:40.797869 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:40.798012 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:40.802239 (+ 4227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:40.802246 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:40.802249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:40.802288 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:40.805212 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:40.805894 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:40.805900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:40.805904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:40.806030 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:40.806390 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:40.806396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:40.806399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:40.806760 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:40.807008 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:40.807325 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 08:30:40.807548 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260928 08:30:40.808553 19494 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41954
I20260928 08:30:40.809793 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 19556
I20260928 08:30:40.825510 19959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:40.810779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47726 (local address 127.15.134.193:41895)
0928 08:30:40.810999 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:40.811006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:40.811225 (+ 219us) server_negotiation.cc:407] Connection header received
0928 08:30:40.811418 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:40.811423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:40.811496 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:40.811596 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:40.812839 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:40.813772 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:40.815691 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:40.816530 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:40.816764 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:40.819966 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:40.819976 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:40.819984 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:40.820031 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:40.823114 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:40.823882 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:40.823886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:40.823888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:40.824027 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:40.824355 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:40.824358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:40.824360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:40.824593 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:40.824835 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:40.825095 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 08:30:40.825280 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 08:30:40.826138 19679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:40.909715 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 19556
I20260928 08:30:40.926771 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 19690
Sep 28 08:30:40 dist-test-slave-qp2n krb5kdc[19450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.194@KRBTEST.COM
I20260928 08:30:40.943137 19964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:40.927591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50128 (local address 127.15.134.194:34447)
0928 08:30:40.928184 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:40.928192 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:40.928217 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:30:40.928274 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:40.928280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:40.928491 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:40.928697 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:40.929922 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:40.930962 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:40.932226 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:40.932894 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:40.933033 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:40.937186 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:40.937198 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:40.937213 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:40.937254 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:40.940139 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:40.940944 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:40.940951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:40.940953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:40.941079 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:40.941491 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:40.941495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:40.941497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:40.941855 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:40.942177 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:40.942516 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 08:30:40.942800 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":343,"threads_started":1}
I20260928 08:30:40.944556 19813 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:41.029995 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 19690
I20260928 08:30:41.049592 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 19823
Sep 28 08:30:41 dist-test-slave-qp2n krb5kdc[19450](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.195@KRBTEST.COM
I20260928 08:30:41.066716 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:41.050799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37814 (local address 127.15.134.195:38007)
0928 08:30:41.051416 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:41.051423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:41.051453 (+ 30us) server_negotiation.cc:407] Connection header received
0928 08:30:41.051514 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:41.051521 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:41.051747 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:41.051883 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:41.053300 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:41.054347 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:41.055765 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:41.056320 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:41.056414 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:41.060767 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:41.060778 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:41.060793 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:41.060830 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:41.063534 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:41.064252 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:41.064259 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:41.064262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:41.064393 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:41.064997 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:41.065002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:41.065005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:41.065410 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:41.065743 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:41.066108 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 08:30:41.066394 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":313,"threads_started":1}
I20260928 08:30:41.067953 19946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:41.153126 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 19823
I20260928 08:30:41.169759 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 19463
I20260928 08:30:41.171126 19525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:41.275588 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 19463
2026-09-28T08:30:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4026 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:30:41 dist-test-slave-qp2n krb5kdc[19971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:30:41 dist-test-slave-qp2n krb5kdc[19971](info): set up 2 sockets
Sep 28 08:30:41 dist-test-slave-qp2n krb5kdc[19971](info): commencing operation
krb5kdc: starting...
W20260928 08:30:42.390954 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:30:42 dist-test-slave-qp2n krb5kdc[19971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:30:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:30:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:30:42.607898 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:42737
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:42737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:42.943127 19984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:42.943413 19984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:42.943578 19984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:42.950760 19984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:30:42.950865 19984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:42.950963 19984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:42.951052 19984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:30:42.951129 19984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:30:42.960999 19984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:42737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:42737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:42.962335 19984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:42.963656 19984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:42.975911 19989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:42.975930 19990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:42.975864 19992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:42.977411 19984 server_base.cc:1034] running on GCE node
I20260928 08:30:42.978310 19984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:42.979831 19984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:42.981036 19984 hybrid_clock.cc:648] HybridClock initialized: now 1790584242980946 us; error 92 us; skew 500 ppm
I20260928 08:30:42.981429 19984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:42 dist-test-slave-qp2n krb5kdc[19971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584242, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:42.984963 19984 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:30:42.986306 19984 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:42.986452 19984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:42.986806 19984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:42.990013 19984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "0ebe31279edc4417bcf232699e8d3c6d"
format_stamp: "Formatted at 2026-09-28 08:30:42 on dist-test-slave-qp2n"
server_key: "64ed4f824a6e245b3ca15845c9231018"
server_key_iv: "d131658fef5a694c119f2bc0bcf5391e"
server_key_version: "encryptionkey@0"
I20260928 08:30:42.990903 19984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "0ebe31279edc4417bcf232699e8d3c6d"
format_stamp: "Formatted at 2026-09-28 08:30:42 on dist-test-slave-qp2n"
server_key: "64ed4f824a6e245b3ca15845c9231018"
server_key_iv: "d131658fef5a694c119f2bc0bcf5391e"
server_key_version: "encryptionkey@0"
I20260928 08:30:42.996599 19984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 08:30:43.000809 19998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:43.002367 19984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:30:43.002528 19984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "0ebe31279edc4417bcf232699e8d3c6d"
format_stamp: "Formatted at 2026-09-28 08:30:42 on dist-test-slave-qp2n"
server_key: "64ed4f824a6e245b3ca15845c9231018"
server_key_iv: "d131658fef5a694c119f2bc0bcf5391e"
server_key_version: "encryptionkey@0"
I20260928 08:30:43.002737 19984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:43.032220 19984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:43.036978 19984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:43.037216 19984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:43.056353 19984 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:42737
I20260928 08:30:43.056375 20049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:42737 every 8 connection(s)
I20260928 08:30:43.057441 19984 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:30:43.058571 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 19984
I20260928 08:30:43.058844 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:30:43.059484 15899 external_mini_cluster.cc:1428] Setting key 4ec765a860440e71168b726fe3093a32
I20260928 08:30:43.061975 20050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 08:30:43 dist-test-slave-qp2n krb5kdc[19971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:43.075690 20050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ebe31279edc4417bcf232699e8d3c6d: Bootstrap starting.
I20260928 08:30:43.079353 20050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ebe31279edc4417bcf232699e8d3c6d: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:43.080562 20050 log.cc:824] T 00000000000000000000000000000000 P 0ebe31279edc4417bcf232699e8d3c6d: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:43.083768 20050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ebe31279edc4417bcf232699e8d3c6d: No bootstrap required, opened a new log
I20260928 08:30:43.085574 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:43.061750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42050 (local address 127.15.134.254:42737)
0928 08:30:43.062610 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:43.062627 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:43.062678 (+ 51us) server_negotiation.cc:407] Connection header received
0928 08:30:43.065074 (+ 2396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:43.065111 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:43.065688 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:43.066154 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:43.067624 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:43.069789 (+ 2165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:43.071182 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:43.072288 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:43.072484 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:43.076926 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:43.076935 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:43.076952 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:43.076994 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:43.080132 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:43.081070 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:43.081077 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:43.081084 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:43.081323 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:43.081926 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:43.081930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:43.081932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:43.082475 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:43.082784 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:43.083262 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 08:30:43.083717 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":389,"threads_started":1}
I20260928 08:30:43.094000 20050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ebe31279edc4417bcf232699e8d3c6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ebe31279edc4417bcf232699e8d3c6d" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 42737 } }
I20260928 08:30:43.094434 20050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ebe31279edc4417bcf232699e8d3c6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:43.094560 20050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ebe31279edc4417bcf232699e8d3c6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ebe31279edc4417bcf232699e8d3c6d, State: Initialized, Role: FOLLOWER
I20260928 08:30:43.095256 20050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ebe31279edc4417bcf232699e8d3c6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ebe31279edc4417bcf232699e8d3c6d" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 42737 } }
I20260928 08:30:43.095505 20050 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ebe31279edc4417bcf232699e8d3c6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:30:43.095642 20050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ebe31279edc4417bcf232699e8d3c6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:30:43.095824 20050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ebe31279edc4417bcf232699e8d3c6d [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:43.098336 20050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ebe31279edc4417bcf232699e8d3c6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ebe31279edc4417bcf232699e8d3c6d" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 42737 } }
I20260928 08:30:43.098896 20050 leader_election.cc:302] T 00000000000000000000000000000000 P 0ebe31279edc4417bcf232699e8d3c6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ebe31279edc4417bcf232699e8d3c6d; no voters:
I20260928 08:30:43.099496 20050 leader_election.cc:288] T 00000000000000000000000000000000 P 0ebe31279edc4417bcf232699e8d3c6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:30:43.099820 20055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ebe31279edc4417bcf232699e8d3c6d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:43.101142 20055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ebe31279edc4417bcf232699e8d3c6d [term 1 LEADER]: Becoming Leader. State: Replica: 0ebe31279edc4417bcf232699e8d3c6d, State: Running, Role: LEADER
I20260928 08:30:43.101742 20055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ebe31279edc4417bcf232699e8d3c6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ebe31279edc4417bcf232699e8d3c6d" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 42737 } }
I20260928 08:30:43.102123 20050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ebe31279edc4417bcf232699e8d3c6d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:30:43.104954 20056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ebe31279edc4417bcf232699e8d3c6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ebe31279edc4417bcf232699e8d3c6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ebe31279edc4417bcf232699e8d3c6d" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 42737 } } }
I20260928 08:30:43.105368 20056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ebe31279edc4417bcf232699e8d3c6d [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:43.105448 20057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ebe31279edc4417bcf232699e8d3c6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ebe31279edc4417bcf232699e8d3c6d. Latest consensus state: current_term: 1 leader_uuid: "0ebe31279edc4417bcf232699e8d3c6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ebe31279edc4417bcf232699e8d3c6d" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 42737 } } }
I20260928 08:30:43.105734 20057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ebe31279edc4417bcf232699e8d3c6d [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:43.107303 20064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:30:43.112164 20064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:30:43.121405 20064 catalog_manager.cc:1347] Generated new cluster ID: 0993387e3951460c8fc7e2257126eef7
I20260928 08:30:43.121539 20064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:30:43.135228 20064 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:30:43.136768 20064 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:30:43.148769 20064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ebe31279edc4417bcf232699e8d3c6d: Generated new TSK 0
I20260928 08:30:43.149657 20064 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.134.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
I20260928 08:30:43.209016 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:0
--local_ip_for_outbound_sockets=127.15.134.193
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:42737
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:43.557381 20076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:43.557695 20076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:43.557870 20076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:43.565503 20076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:43.565631 20076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:43.565819 20076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:30:43.576460 20076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:42737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:43.577914 20076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:43.579259 20076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:43.592469 20084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:43.592552 20081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:43.592506 20082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:43.598536 20076 server_base.cc:1034] running on GCE node
I20260928 08:30:43.599203 20076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:43.600394 20076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:43.601712 20076 hybrid_clock.cc:648] HybridClock initialized: now 1790584243601693 us; error 81 us; skew 500 ppm
I20260928 08:30:43.602099 20076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:43 dist-test-slave-qp2n krb5kdc[19971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584243, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:43.606032 20076 init.cc:377] Logged in from keytab as kudu/127.15.134.193@KRBTEST.COM (short username kudu)
I20260928 08:30:43.607798 20076 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:43.607939 20076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:43.608274 20076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:43.611481 20076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-0/data/instance:
uuid: "c078080da9644522a981ed406e91bc03"
format_stamp: "Formatted at 2026-09-28 08:30:43 on dist-test-slave-qp2n"
server_key: "7022b21fcdc97db1d5aac563127baa93"
server_key_iv: "fb59e89ea93ebd3b32298745c08b875d"
server_key_version: "encryptionkey@0"
I20260928 08:30:43.612295 20076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance:
uuid: "c078080da9644522a981ed406e91bc03"
format_stamp: "Formatted at 2026-09-28 08:30:43 on dist-test-slave-qp2n"
server_key: "7022b21fcdc97db1d5aac563127baa93"
server_key_iv: "fb59e89ea93ebd3b32298745c08b875d"
server_key_version: "encryptionkey@0"
I20260928 08:30:43.617476 20076 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 08:30:43.620929 20090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:43.622203 20076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:30:43.622386 20076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "c078080da9644522a981ed406e91bc03"
format_stamp: "Formatted at 2026-09-28 08:30:43 on dist-test-slave-qp2n"
server_key: "7022b21fcdc97db1d5aac563127baa93"
server_key_iv: "fb59e89ea93ebd3b32298745c08b875d"
server_key_version: "encryptionkey@0"
I20260928 08:30:43.622604 20076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:43.641003 20076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:43.645327 20076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:43.645607 20076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:43.646570 20076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:30:43.648066 20076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:43.648196 20076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:43.648331 20076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:43.648419 20076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:43.683101 20076 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:36397
I20260928 08:30:43.683190 20202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:36397 every 8 connection(s)
I20260928 08:30:43.684001 20076 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:30:43.692304 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 20076
I20260928 08:30:43.692545 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance
I20260928 08:30:43.693243 15899 external_mini_cluster.cc:1428] Setting key 5a089835e7e3579bff80ef49385180b9
Sep 28 08:30:43 dist-test-slave-qp2n krb5kdc[19971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584243, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:43.706738 20205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:43.686693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:33453 (local address 127.15.134.254:42737)
0928 08:30:43.687156 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:43.687161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:43.688115 (+ 954us) server_negotiation.cc:407] Connection header received
0928 08:30:43.690097 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:43.690102 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:43.690202 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:43.690323 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:43.691906 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:43.692820 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:43.694660 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:43.695211 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:43.695355 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:43.699866 (+ 4511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:43.699871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:43.699875 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:43.699911 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:43.703220 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:43.704052 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:43.704058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:43.704062 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:43.704198 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:43.704674 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:43.704680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:43.704684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:43.705054 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:43.705247 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:43.706086 (+ 839us) server_negotiation.cc:299] Negotiation successful
0928 08:30:43.706344 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":254,"threads_started":1}
I20260928 08:30:43.707906 20206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:43.687207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:42737 (local address 127.15.134.193:33453)
0928 08:30:43.687934 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:43.687980 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:43.689819 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:43.690567 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:43.690577 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:43.691110 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:43.691729 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:43.691748 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:43.692985 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:43.692991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:43.694501 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:43.694514 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:43.695479 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:43.695487 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:43.695868 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:43.696714 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:43.696752 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:43.699598 (+ 2846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:43.703397 (+ 3799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:43.703408 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:43.703429 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:43.703909 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:43.704317 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:43.704322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:43.704325 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:43.704529 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:43.705230 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:43.705239 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:43.705649 (+ 410us) client_negotiation.cc:769] Sending connection context
0928 08:30:43.705972 (+ 323us) client_negotiation.cc:240] Negotiation successful
0928 08:30:43.706654 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":275,"threads_started":1}
I20260928 08:30:43.709725 20203 heartbeater.cc:347] Connected to a master server at 127.15.134.254:42737
I20260928 08:30:43.710109 20203 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:43.711083 20203 heartbeater.cc:511] Master 127.15.134.254:42737 requested a full tablet report, sending...
I20260928 08:30:43.714174 20015 ts_manager.cc:194] Registered new tserver with Master: c078080da9644522a981ed406e91bc03 (127.15.134.193:36397)
WARNING: no policy specified for kudu/127.15.134.194@KRBTEST.COM; defaulting to no policy
I20260928 08:30:43.716042 20015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.193@KRBTEST.COM'} at 127.15.134.193:33453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
I20260928 08:30:43.744242 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:0
--local_ip_for_outbound_sockets=127.15.134.194
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:42737
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:44.076440 20209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:44.076839 20209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:44.076946 20209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:44.084375 20209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:44.084486 20209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:44.084601 20209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:30:44.094682 20209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:42737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:44.096125 20209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:44.097522 20209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:44.110944 20215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:44.111052 20214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:44.111073 20217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:44.112149 20209 server_base.cc:1034] running on GCE node
I20260928 08:30:44.112792 20209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:44.113735 20209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:44.114935 20209 hybrid_clock.cc:648] HybridClock initialized: now 1790584244114893 us; error 42 us; skew 500 ppm
I20260928 08:30:44.115254 20209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:44 dist-test-slave-qp2n krb5kdc[19971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584244, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:44.121086 20209 init.cc:377] Logged in from keytab as kudu/127.15.134.194@KRBTEST.COM (short username kudu)
I20260928 08:30:44.122243 20209 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:44.122367 20209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:44.122695 20209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:44.125767 20209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-1/data/instance:
uuid: "8ccaf52c25044b2b97a6a34f34a8fe93"
format_stamp: "Formatted at 2026-09-28 08:30:44 on dist-test-slave-qp2n"
server_key: "16c0084a86ea20958b4262bed3e40521"
server_key_iv: "7ed4fdaf69e8657d1bb795c70b0bb6d7"
server_key_version: "encryptionkey@0"
I20260928 08:30:44.126600 20209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance:
uuid: "8ccaf52c25044b2b97a6a34f34a8fe93"
format_stamp: "Formatted at 2026-09-28 08:30:44 on dist-test-slave-qp2n"
server_key: "16c0084a86ea20958b4262bed3e40521"
server_key_iv: "7ed4fdaf69e8657d1bb795c70b0bb6d7"
server_key_version: "encryptionkey@0"
I20260928 08:30:44.131603 20209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 08:30:44.135205 20223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:44.136566 20209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:30:44.136802 20209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "8ccaf52c25044b2b97a6a34f34a8fe93"
format_stamp: "Formatted at 2026-09-28 08:30:44 on dist-test-slave-qp2n"
server_key: "16c0084a86ea20958b4262bed3e40521"
server_key_iv: "7ed4fdaf69e8657d1bb795c70b0bb6d7"
server_key_version: "encryptionkey@0"
I20260928 08:30:44.137029 20209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:44.163019 20209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:44.167518 20209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:44.167840 20209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:44.169015 20209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:30:44.170477 20209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:44.170601 20209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:44.170745 20209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:44.170841 20209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:44.206174 20209 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:35841
I20260928 08:30:44.206310 20335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:35841 every 8 connection(s)
I20260928 08:30:44.207077 20209 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:30:44.215099 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 20209
I20260928 08:30:44.215332 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance
I20260928 08:30:44.215754 15899 external_mini_cluster.cc:1428] Setting key 3cea2260acc00abfa1684894f9ce2f0b
Sep 28 08:30:44 dist-test-slave-qp2n krb5kdc[19971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584244, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:44.229676 20338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:44.209827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:59659 (local address 127.15.134.254:42737)
0928 08:30:44.210288 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:44.210292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:44.211274 (+ 982us) server_negotiation.cc:407] Connection header received
0928 08:30:44.213167 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:44.213173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:44.213263 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:44.213383 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:44.214923 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:44.215749 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:44.217512 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:44.217967 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:44.218108 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:44.222181 (+ 4073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:44.222187 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:44.222191 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:44.222238 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:44.226089 (+ 3851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:44.226869 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:44.226877 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:44.226883 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:44.227038 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:44.227562 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:44.227569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:44.227575 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:44.227963 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:44.228115 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:44.228867 (+ 752us) server_negotiation.cc:299] Negotiation successful
0928 08:30:44.229140 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":251,"threads_started":1}
I20260928 08:30:44.230317 20339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:44.210237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:42737 (local address 127.15.134.194:59659)
0928 08:30:44.210991 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:44.211032 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:44.212897 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:44.213621 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:44.213629 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:44.214165 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:44.214716 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:44.214735 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:44.215899 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:44.215906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:44.217366 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:44.217379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:44.218194 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:44.218201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:44.218423 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:44.219194 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:44.219218 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:44.221903 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:44.226280 (+ 4377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:44.226288 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:44.226303 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:44.226724 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:44.227161 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:44.227165 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:44.227167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:44.227430 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:44.228080 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:44.228086 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:44.228438 (+ 352us) client_negotiation.cc:769] Sending connection context
0928 08:30:44.228729 (+ 291us) client_negotiation.cc:240] Negotiation successful
0928 08:30:44.229144 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":335,"threads_started":1}
I20260928 08:30:44.231894 20336 heartbeater.cc:347] Connected to a master server at 127.15.134.254:42737
I20260928 08:30:44.232224 20336 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:44.233026 20336 heartbeater.cc:511] Master 127.15.134.254:42737 requested a full tablet report, sending...
I20260928 08:30:44.234531 20015 ts_manager.cc:194] Registered new tserver with Master: 8ccaf52c25044b2b97a6a34f34a8fe93 (127.15.134.194:35841)
I20260928 08:30:44.235368 20015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.194@KRBTEST.COM'} at 127.15.134.194:59659
WARNING: no policy specified for kudu/127.15.134.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
I20260928 08:30:44.264937 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:0
--local_ip_for_outbound_sockets=127.15.134.195
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:42737
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:44.620800 20342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:44.621093 20342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:44.621274 20342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:44.629858 20342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:44.630013 20342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:44.630235 20342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:30:44.641537 20342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:42737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:44.643121 20342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:44.644595 20342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:44.658779 20347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:44.658823 20348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:44.658793 20350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:44.659543 20342 server_base.cc:1034] running on GCE node
I20260928 08:30:44.660230 20342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:44.661391 20342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:44.662633 20342 hybrid_clock.cc:648] HybridClock initialized: now 1790584244662591 us; error 58 us; skew 500 ppm
I20260928 08:30:44.662987 20342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:44 dist-test-slave-qp2n krb5kdc[19971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584244, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:44.666481 20342 init.cc:377] Logged in from keytab as kudu/127.15.134.195@KRBTEST.COM (short username kudu)
I20260928 08:30:44.667846 20342 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:44.667986 20342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:44.668310 20342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:44.671387 20342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-2/data/instance:
uuid: "9fae13b8fd8946ac82b9afee8651d5e5"
format_stamp: "Formatted at 2026-09-28 08:30:44 on dist-test-slave-qp2n"
server_key: "b89fcef9dc8c99319c5c295e9f891fd2"
server_key_iv: "516f48a88f73e7140eae3f81890278c4"
server_key_version: "encryptionkey@0"
I20260928 08:30:44.672258 20342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance:
uuid: "9fae13b8fd8946ac82b9afee8651d5e5"
format_stamp: "Formatted at 2026-09-28 08:30:44 on dist-test-slave-qp2n"
server_key: "b89fcef9dc8c99319c5c295e9f891fd2"
server_key_iv: "516f48a88f73e7140eae3f81890278c4"
server_key_version: "encryptionkey@0"
I20260928 08:30:44.677732 20342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 08:30:44.681499 20356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:44.682857 20342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:30:44.683044 20342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "9fae13b8fd8946ac82b9afee8651d5e5"
format_stamp: "Formatted at 2026-09-28 08:30:44 on dist-test-slave-qp2n"
server_key: "b89fcef9dc8c99319c5c295e9f891fd2"
server_key_iv: "516f48a88f73e7140eae3f81890278c4"
server_key_version: "encryptionkey@0"
I20260928 08:30:44.683283 20342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:44.718523 20342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:44.719661 20203 heartbeater.cc:503] Master 127.15.134.254:42737 was elected leader, sending a full tablet report...
I20260928 08:30:44.723601 20342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:44.723843 20342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:44.724821 20342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:30:44.726400 20342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:44.726488 20342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:44.726562 20342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:44.726640 20342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:44.761920 20342 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:46147
I20260928 08:30:44.761963 20468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:46147 every 8 connection(s)
I20260928 08:30:44.762814 20342 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:30:44.766527 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 20342
I20260928 08:30:44.766801 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance
I20260928 08:30:44.767335 15899 external_mini_cluster.cc:1428] Setting key 92b5e4d3f6a6b31bb6760374b5a335f8
Sep 28 08:30:44 dist-test-slave-qp2n krb5kdc[19971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584244, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:44.785141 20471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:44.765445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:36399 (local address 127.15.134.254:42737)
0928 08:30:44.766015 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:44.766020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:44.767065 (+ 1045us) server_negotiation.cc:407] Connection header received
0928 08:30:44.769450 (+ 2385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:44.769454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:44.769539 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:44.769624 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:44.771423 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:44.772256 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:44.773905 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:44.774337 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:44.774433 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:44.778055 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:44.778060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:44.778063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:44.778104 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:44.781385 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:44.782469 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:44.782474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:44.782476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:44.782750 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:44.783337 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:44.783340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:44.783342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:44.783660 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:44.783813 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:44.784563 (+ 750us) server_negotiation.cc:299] Negotiation successful
0928 08:30:44.784853 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":350,"threads_started":1}
I20260928 08:30:44.786126 20472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:44.765869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:42737 (local address 127.15.134.195:36399)
0928 08:30:44.766744 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:44.766822 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:44.769135 (+ 2313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:44.769890 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:44.769902 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:44.770635 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:44.771236 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:44.771253 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:44.772398 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:44.772401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:44.773783 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:44.773796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:44.774561 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:44.774567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:44.774787 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:44.775449 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:44.775472 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:44.777806 (+ 2334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:44.781567 (+ 3761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:44.781576 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:44.781591 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:44.782319 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:44.782922 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:44.782930 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:44.782937 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:44.783202 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:44.783814 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:44.783820 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:44.784171 (+ 351us) client_negotiation.cc:769] Sending connection context
0928 08:30:44.784420 (+ 249us) client_negotiation.cc:240] Negotiation successful
0928 08:30:44.784952 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":335,"threads_started":1}
I20260928 08:30:44.788095 20469 heartbeater.cc:347] Connected to a master server at 127.15.134.254:42737
I20260928 08:30:44.788431 20469 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:44.789206 20469 heartbeater.cc:511] Master 127.15.134.254:42737 requested a full tablet report, sending...
I20260928 08:30:44.790695 20015 ts_manager.cc:194] Registered new tserver with Master: 9fae13b8fd8946ac82b9afee8651d5e5 (127.15.134.195:46147)
I20260928 08:30:44.791510 20015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.195@KRBTEST.COM'} at 127.15.134.195:36399
I20260928 08:30:44.795814 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:30:44.813716 20471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:44.800683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42060 (local address 127.15.134.254:42737)
0928 08:30:44.800887 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:44.800891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:44.801433 (+ 542us) server_negotiation.cc:407] Connection header received
0928 08:30:44.801682 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:44.801687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:44.801750 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:44.801847 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:44.803508 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:44.804252 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:44.805687 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:44.806185 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:44.806312 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:44.808693 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:44.808702 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:44.808711 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:44.808743 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:44.811164 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:44.811846 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:44.811850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:44.811852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:44.811963 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:44.812269 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:44.812272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:44.812274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:44.812655 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:44.812884 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:44.813209 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 08:30:44.813404 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 08:30:44.818143 20015 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:30:44.825768 20015 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:30:44.864408 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:44.839800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:36397 (local address 127.0.0.1:45742)
0928 08:30:44.855130 (+ 15330us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:44.855162 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:44.855364 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:44.857181 (+ 1817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:44.857189 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:44.857214 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:30:44.857640 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:44.857653 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:44.859509 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:44.859516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:44.861824 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:44.861836 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:44.863337 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:44.863343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:44.863542 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:44.863612 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 08:30:44.863741 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 08:30:44.863940 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15102,"mutex_wait_us":49,"thread_start_us":16171,"threads_started":1}
I20260928 08:30:44.865665 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:44.839932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45742 (local address 127.15.134.193:36397)
0928 08:30:44.856024 (+ 16092us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:44.856046 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:44.856082 (+ 36us) server_negotiation.cc:407] Connection header received
0928 08:30:44.856195 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:44.856205 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:44.856544 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:44.856828 (+ 284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:30:44.857838 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:44.859382 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:44.862000 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:44.863116 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:44.863361 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:44.865055 (+ 1694us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:44.865133 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 08:30:44.865325 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15903,"thread_start_us":15757,"threads_started":1}
I20260928 08:30:44.866007 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:44.839823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53166 (local address 127.15.134.194:35841)
0928 08:30:44.858145 (+ 18322us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:44.858154 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:44.858178 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:30:44.858275 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:44.858283 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:44.858524 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:44.858733 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:30:44.859292 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:44.860805 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:44.863426 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:44.864922 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:44.865088 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:44.865258 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:44.865549 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 08:30:44.865715 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18150,"thread_start_us":18009,"threads_started":1}
I20260928 08:30:44.866130 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:44.839328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:35841 (local address 127.0.0.1:53166)
0928 08:30:44.852329 (+ 13001us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:44.852391 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:44.852687 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:44.858754 (+ 6067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:44.858759 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:44.858787 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:30:44.859086 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:44.859095 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:44.860978 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:44.860984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:44.863226 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:44.863241 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:44.865184 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:44.865191 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:44.865374 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:44.865396 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 08:30:44.865506 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 08:30:44.865647 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12689,"thread_start_us":12509,"threads_started":1}
I20260928 08:30:44.867045 20486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:44.840179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:46147 (local address 127.0.0.1:59006)
0928 08:30:44.857083 (+ 16904us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:44.857104 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:44.857214 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:44.858133 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:44.858139 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:44.858169 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:30:44.858588 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:44.858602 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:44.860604 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:44.860631 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:44.863057 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:44.863077 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:44.865257 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:44.865263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:44.865450 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:44.865485 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 08:30:44.865589 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 08:30:44.865741 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16771,"mutex_wait_us":3,"thread_start_us":16605,"threads_started":1}
I20260928 08:30:44.867735 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:44.840897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59006 (local address 127.15.134.195:46147)
0928 08:30:44.857452 (+ 16555us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:44.857462 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:44.857495 (+ 33us) server_negotiation.cc:407] Connection header received
0928 08:30:44.857619 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:44.857628 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:44.857882 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:44.858088 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:30:44.858822 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:44.860424 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:44.863250 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:44.864764 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:44.864951 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:44.865104 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:44.867205 (+ 2101us) server_negotiation.cc:299] Negotiation successful
0928 08:30:44.867366 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15578,"thread_start_us":15450,"threads_started":1}
I20260928 08:30:44.869880 20271 tablet_service.cc:1467] Processing CreateTablet for tablet 27226749529b464fb5097bc5677aedf1 (DEFAULT_TABLE table=test-table [id=1530838ae00040df9507060cc076c1c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:44.869892 20138 tablet_service.cc:1467] Processing CreateTablet for tablet 27226749529b464fb5097bc5677aedf1 (DEFAULT_TABLE table=test-table [id=1530838ae00040df9507060cc076c1c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:44.871152 20271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27226749529b464fb5097bc5677aedf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:44.871291 20138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27226749529b464fb5097bc5677aedf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:44.871430 20404 tablet_service.cc:1467] Processing CreateTablet for tablet 27226749529b464fb5097bc5677aedf1 (DEFAULT_TABLE table=test-table [id=1530838ae00040df9507060cc076c1c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:44.872819 20404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27226749529b464fb5097bc5677aedf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:44.882956 20488 tablet_bootstrap.cc:492] T 27226749529b464fb5097bc5677aedf1 P 8ccaf52c25044b2b97a6a34f34a8fe93: Bootstrap starting.
I20260928 08:30:44.885823 20489 tablet_bootstrap.cc:492] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03: Bootstrap starting.
I20260928 08:30:44.886839 20488 tablet_bootstrap.cc:654] T 27226749529b464fb5097bc5677aedf1 P 8ccaf52c25044b2b97a6a34f34a8fe93: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:44.888904 20488 log.cc:824] T 27226749529b464fb5097bc5677aedf1 P 8ccaf52c25044b2b97a6a34f34a8fe93: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:44.889385 20490 tablet_bootstrap.cc:492] T 27226749529b464fb5097bc5677aedf1 P 9fae13b8fd8946ac82b9afee8651d5e5: Bootstrap starting.
I20260928 08:30:44.889485 20489 tablet_bootstrap.cc:654] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:44.890664 20489 log.cc:824] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:44.892735 20488 tablet_bootstrap.cc:492] T 27226749529b464fb5097bc5677aedf1 P 8ccaf52c25044b2b97a6a34f34a8fe93: No bootstrap required, opened a new log
I20260928 08:30:44.892970 20490 tablet_bootstrap.cc:654] T 27226749529b464fb5097bc5677aedf1 P 9fae13b8fd8946ac82b9afee8651d5e5: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:44.893031 20488 ts_tablet_manager.cc:1397] T 27226749529b464fb5097bc5677aedf1 P 8ccaf52c25044b2b97a6a34f34a8fe93: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260928 08:30:44.893909 20489 tablet_bootstrap.cc:492] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03: No bootstrap required, opened a new log
I20260928 08:30:44.893960 20490 log.cc:824] T 27226749529b464fb5097bc5677aedf1 P 9fae13b8fd8946ac82b9afee8651d5e5: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:44.894194 20489 ts_tablet_manager.cc:1397] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260928 08:30:44.896845 20490 tablet_bootstrap.cc:492] T 27226749529b464fb5097bc5677aedf1 P 9fae13b8fd8946ac82b9afee8651d5e5: No bootstrap required, opened a new log
I20260928 08:30:44.897119 20490 ts_tablet_manager.cc:1397] T 27226749529b464fb5097bc5677aedf1 P 9fae13b8fd8946ac82b9afee8651d5e5: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 08:30:44.904578 20488 raft_consensus.cc:348] T 27226749529b464fb5097bc5677aedf1 P 8ccaf52c25044b2b97a6a34f34a8fe93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccaf52c25044b2b97a6a34f34a8fe93" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35841 } } peers { permanent_uuid: "c078080da9644522a981ed406e91bc03" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 36397 } } peers { permanent_uuid: "9fae13b8fd8946ac82b9afee8651d5e5" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 46147 } }
I20260928 08:30:44.905076 20488 raft_consensus.cc:374] T 27226749529b464fb5097bc5677aedf1 P 8ccaf52c25044b2b97a6a34f34a8fe93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:44.904793 20489 raft_consensus.cc:348] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccaf52c25044b2b97a6a34f34a8fe93" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35841 } } peers { permanent_uuid: "c078080da9644522a981ed406e91bc03" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 36397 } } peers { permanent_uuid: "9fae13b8fd8946ac82b9afee8651d5e5" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 46147 } }
I20260928 08:30:44.904855 20490 raft_consensus.cc:348] T 27226749529b464fb5097bc5677aedf1 P 9fae13b8fd8946ac82b9afee8651d5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccaf52c25044b2b97a6a34f34a8fe93" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35841 } } peers { permanent_uuid: "c078080da9644522a981ed406e91bc03" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 36397 } } peers { permanent_uuid: "9fae13b8fd8946ac82b9afee8651d5e5" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 46147 } }
I20260928 08:30:44.905211 20489 raft_consensus.cc:374] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:44.905176 20488 raft_consensus.cc:729] T 27226749529b464fb5097bc5677aedf1 P 8ccaf52c25044b2b97a6a34f34a8fe93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ccaf52c25044b2b97a6a34f34a8fe93, State: Initialized, Role: FOLLOWER
I20260928 08:30:44.905316 20489 raft_consensus.cc:729] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c078080da9644522a981ed406e91bc03, State: Initialized, Role: FOLLOWER
I20260928 08:30:44.905444 20490 raft_consensus.cc:374] T 27226749529b464fb5097bc5677aedf1 P 9fae13b8fd8946ac82b9afee8651d5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:44.905575 20490 raft_consensus.cc:729] T 27226749529b464fb5097bc5677aedf1 P 9fae13b8fd8946ac82b9afee8651d5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fae13b8fd8946ac82b9afee8651d5e5, State: Initialized, Role: FOLLOWER
I20260928 08:30:44.905894 20488 consensus_queue.cc:260] T 27226749529b464fb5097bc5677aedf1 P 8ccaf52c25044b2b97a6a34f34a8fe93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccaf52c25044b2b97a6a34f34a8fe93" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35841 } } peers { permanent_uuid: "c078080da9644522a981ed406e91bc03" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 36397 } } peers { permanent_uuid: "9fae13b8fd8946ac82b9afee8651d5e5" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 46147 } }
I20260928 08:30:44.905905 20489 consensus_queue.cc:260] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccaf52c25044b2b97a6a34f34a8fe93" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35841 } } peers { permanent_uuid: "c078080da9644522a981ed406e91bc03" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 36397 } } peers { permanent_uuid: "9fae13b8fd8946ac82b9afee8651d5e5" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 46147 } }
I20260928 08:30:44.906298 20490 consensus_queue.cc:260] T 27226749529b464fb5097bc5677aedf1 P 9fae13b8fd8946ac82b9afee8651d5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccaf52c25044b2b97a6a34f34a8fe93" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35841 } } peers { permanent_uuid: "c078080da9644522a981ed406e91bc03" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 36397 } } peers { permanent_uuid: "9fae13b8fd8946ac82b9afee8651d5e5" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 46147 } }
I20260928 08:30:44.908437 20490 ts_tablet_manager.cc:1428] T 27226749529b464fb5097bc5677aedf1 P 9fae13b8fd8946ac82b9afee8651d5e5: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260928 08:30:44.908905 20469 heartbeater.cc:503] Master 127.15.134.254:42737 was elected leader, sending a full tablet report...
I20260928 08:30:44.913865 20488 ts_tablet_manager.cc:1428] T 27226749529b464fb5097bc5677aedf1 P 8ccaf52c25044b2b97a6a34f34a8fe93: Time spent starting tablet: real 0.021s user 0.012s sys 0.008s
I20260928 08:30:44.914906 20489 ts_tablet_manager.cc:1428] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03: Time spent starting tablet: real 0.021s user 0.017s sys 0.003s
I20260928 08:30:44.915400 20336 heartbeater.cc:503] Master 127.15.134.254:42737 was elected leader, sending a full tablet report...
W20260928 08:30:44.936527 20204 tablet.cc:2369] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:30:44.959141 20337 tablet.cc:2369] T 27226749529b464fb5097bc5677aedf1 P 8ccaf52c25044b2b97a6a34f34a8fe93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:30:44.991468 20494 raft_consensus.cc:482] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:30:44.991955 20494 raft_consensus.cc:504] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccaf52c25044b2b97a6a34f34a8fe93" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35841 } } peers { permanent_uuid: "c078080da9644522a981ed406e91bc03" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 36397 } } peers { permanent_uuid: "9fae13b8fd8946ac82b9afee8651d5e5" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 46147 } }
I20260928 08:30:44.993849 20494 leader_election.cc:288] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ccaf52c25044b2b97a6a34f34a8fe93 (127.15.134.194:35841), 9fae13b8fd8946ac82b9afee8651d5e5 (127.15.134.195:46147)
I20260928 08:30:45.001029 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:44.994231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:51359 (local address 127.15.134.194:35841)
0928 08:30:44.994566 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:44.994576 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:44.995360 (+ 784us) server_negotiation.cc:407] Connection header received
0928 08:30:44.995569 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:44.995574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:44.995645 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:44.995758 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:30:44.996385 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:44.997171 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:44.999299 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:45.000033 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:45.000228 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:45.000308 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:45.000593 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 08:30:45.000743 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164}
I20260928 08:30:45.001060 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:44.994404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:35671 (local address 127.15.134.195:46147)
0928 08:30:44.994667 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:44.994677 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:44.995305 (+ 628us) server_negotiation.cc:407] Connection header received
0928 08:30:44.995536 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:44.995540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:44.995599 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:44.995697 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:30:44.996323 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:44.997121 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:44.999296 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:45.000068 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:45.000263 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:45.000346 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:45.000671 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 08:30:45.000799 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:30:45.001185 20497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:44.994288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:35841 (local address 127.15.134.193:51359)
0928 08:30:44.995250 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:44.995265 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:44.995426 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:44.995773 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:44.995777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:44.995799 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:30:44.996199 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:44.996207 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:44.997357 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:44.997361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:44.999146 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:44.999158 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:45.000266 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:45.000273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:45.000426 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:45.000453 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 08:30:45.000557 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 08:30:45.000742 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":5,"thread_start_us":520,"threads_started":1}
I20260928 08:30:45.001442 20498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:44.994266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:46147 (local address 127.15.134.193:35671)
0928 08:30:44.995178 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:44.995195 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:44.995384 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:44.995733 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:44.995737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:44.995765 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:30:44.996170 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:44.996179 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:44.997276 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:44.997285 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:44.999131 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:44.999144 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:45.000314 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:45.000320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:45.000495 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:45.000516 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 08:30:45.000622 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 08:30:45.000791 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":4,"thread_start_us":669,"threads_started":1}
I20260928 08:30:45.002342 20291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27226749529b464fb5097bc5677aedf1" candidate_uuid: "c078080da9644522a981ed406e91bc03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccaf52c25044b2b97a6a34f34a8fe93" is_pre_election: true
I20260928 08:30:45.002470 20424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27226749529b464fb5097bc5677aedf1" candidate_uuid: "c078080da9644522a981ed406e91bc03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fae13b8fd8946ac82b9afee8651d5e5" is_pre_election: true
I20260928 08:30:45.002883 20291 raft_consensus.cc:2454] T 27226749529b464fb5097bc5677aedf1 P 8ccaf52c25044b2b97a6a34f34a8fe93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c078080da9644522a981ed406e91bc03 in term 0.
I20260928 08:30:45.002996 20424 raft_consensus.cc:2454] T 27226749529b464fb5097bc5677aedf1 P 9fae13b8fd8946ac82b9afee8651d5e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c078080da9644522a981ed406e91bc03 in term 0.
I20260928 08:30:45.003644 20094 leader_election.cc:302] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fae13b8fd8946ac82b9afee8651d5e5, c078080da9644522a981ed406e91bc03; no voters:
I20260928 08:30:45.004073 20494 raft_consensus.cc:2781] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:30:45.004215 20494 raft_consensus.cc:482] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:30:45.004343 20494 raft_consensus.cc:3037] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:45.006865 20494 raft_consensus.cc:504] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccaf52c25044b2b97a6a34f34a8fe93" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35841 } } peers { permanent_uuid: "c078080da9644522a981ed406e91bc03" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 36397 } } peers { permanent_uuid: "9fae13b8fd8946ac82b9afee8651d5e5" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 46147 } }
I20260928 08:30:45.007545 20494 leader_election.cc:288] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03 [CANDIDATE]: Term 1 election: Requested vote from peers 8ccaf52c25044b2b97a6a34f34a8fe93 (127.15.134.194:35841), 9fae13b8fd8946ac82b9afee8651d5e5 (127.15.134.195:46147)
I20260928 08:30:45.008059 20291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27226749529b464fb5097bc5677aedf1" candidate_uuid: "c078080da9644522a981ed406e91bc03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ccaf52c25044b2b97a6a34f34a8fe93"
I20260928 08:30:45.008083 20424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27226749529b464fb5097bc5677aedf1" candidate_uuid: "c078080da9644522a981ed406e91bc03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fae13b8fd8946ac82b9afee8651d5e5"
I20260928 08:30:45.008273 20291 raft_consensus.cc:3037] T 27226749529b464fb5097bc5677aedf1 P 8ccaf52c25044b2b97a6a34f34a8fe93 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:45.008292 20424 raft_consensus.cc:3037] T 27226749529b464fb5097bc5677aedf1 P 9fae13b8fd8946ac82b9afee8651d5e5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:45.010521 20424 raft_consensus.cc:2454] T 27226749529b464fb5097bc5677aedf1 P 9fae13b8fd8946ac82b9afee8651d5e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c078080da9644522a981ed406e91bc03 in term 1.
I20260928 08:30:45.010528 20291 raft_consensus.cc:2454] T 27226749529b464fb5097bc5677aedf1 P 8ccaf52c25044b2b97a6a34f34a8fe93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c078080da9644522a981ed406e91bc03 in term 1.
I20260928 08:30:45.010962 20094 leader_election.cc:302] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9fae13b8fd8946ac82b9afee8651d5e5, c078080da9644522a981ed406e91bc03; no voters:
I20260928 08:30:45.011266 20494 raft_consensus.cc:2781] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:45.011807 20494 raft_consensus.cc:686] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03 [term 1 LEADER]: Becoming Leader. State: Replica: c078080da9644522a981ed406e91bc03, State: Running, Role: LEADER
I20260928 08:30:45.012501 20494 consensus_queue.cc:237] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccaf52c25044b2b97a6a34f34a8fe93" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35841 } } peers { permanent_uuid: "c078080da9644522a981ed406e91bc03" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 36397 } } peers { permanent_uuid: "9fae13b8fd8946ac82b9afee8651d5e5" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 46147 } }
W20260928 08:30:45.014256 20470 tablet.cc:2369] T 27226749529b464fb5097bc5677aedf1 P 9fae13b8fd8946ac82b9afee8651d5e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:30:45.017149 20014 catalog_manager.cc:5495] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03 reported cstate change: term changed from 0 to 1, leader changed from <none> to c078080da9644522a981ed406e91bc03 (127.15.134.193). New cstate: current_term: 1 leader_uuid: "c078080da9644522a981ed406e91bc03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ccaf52c25044b2b97a6a34f34a8fe93" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c078080da9644522a981ed406e91bc03" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 36397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fae13b8fd8946ac82b9afee8651d5e5" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 46147 } health_report { overall_health: UNKNOWN } } }
I20260928 08:30:45.062913 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:45.057021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45746 (local address 127.15.134.193:36397)
0928 08:30:45.057282 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:45.057288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:45.057493 (+ 205us) server_negotiation.cc:407] Connection header received
0928 08:30:45.057763 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:45.057767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:45.057848 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:45.058014 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:30:45.058404 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:45.059148 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:45.060547 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:45.061038 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:45.061190 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:45.061730 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:30:45.062292 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:30:45.062463 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:45.062546 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 08:30:45.062636 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 08:30:45.077775 20424 raft_consensus.cc:1261] T 27226749529b464fb5097bc5677aedf1 P 9fae13b8fd8946ac82b9afee8651d5e5 [term 1 FOLLOWER]: Refusing update from remote peer c078080da9644522a981ed406e91bc03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:30:45.078052 20291 raft_consensus.cc:1261] T 27226749529b464fb5097bc5677aedf1 P 8ccaf52c25044b2b97a6a34f34a8fe93 [term 1 FOLLOWER]: Refusing update from remote peer c078080da9644522a981ed406e91bc03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:30:45.079061 20499 consensus_queue.cc:1035] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fae13b8fd8946ac82b9afee8651d5e5" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 46147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260928 08:30:45.080437 20494 consensus_queue.cc:1035] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ccaf52c25044b2b97a6a34f34a8fe93" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:30:45.109491 20014 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:30:45.109815 20014 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:30:45.114221 20014 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0ebe31279edc4417bcf232699e8d3c6d: Sending DeleteTablet for 3 replicas of tablet 27226749529b464fb5097bc5677aedf1
I20260928 08:30:45.115639 20271 tablet_service.cc:1514] Processing DeleteTablet for tablet 27226749529b464fb5097bc5677aedf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:30:45 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:53166
I20260928 08:30:45.115661 20138 tablet_service.cc:1514] Processing DeleteTablet for tablet 27226749529b464fb5097bc5677aedf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:30:45 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:45742
I20260928 08:30:45.116240 20404 tablet_service.cc:1514] Processing DeleteTablet for tablet 27226749529b464fb5097bc5677aedf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:30:45 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:59006
I20260928 08:30:45.120194 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 20076
I20260928 08:30:45.131145 20513 tablet_replica.cc:330] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03: stopping tablet replica
I20260928 08:30:45.131574 20512 tablet_replica.cc:330] T 27226749529b464fb5097bc5677aedf1 P 8ccaf52c25044b2b97a6a34f34a8fe93: stopping tablet replica
I20260928 08:30:45.131692 20513 raft_consensus.cc:2229] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:30:45.131986 20512 raft_consensus.cc:2229] T 27226749529b464fb5097bc5677aedf1 P 8ccaf52c25044b2b97a6a34f34a8fe93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:30:45.132416 20513 raft_consensus.cc:2258] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:30:45.132535 20512 raft_consensus.cc:2258] T 27226749529b464fb5097bc5677aedf1 P 8ccaf52c25044b2b97a6a34f34a8fe93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:30:45.134482 20514 tablet_replica.cc:330] T 27226749529b464fb5097bc5677aedf1 P 9fae13b8fd8946ac82b9afee8651d5e5: stopping tablet replica
I20260928 08:30:45.134749 20513 ts_tablet_manager.cc:1905] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:30:45.134742 20512 ts_tablet_manager.cc:1905] T 27226749529b464fb5097bc5677aedf1 P 8ccaf52c25044b2b97a6a34f34a8fe93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:30:45.135006 20514 raft_consensus.cc:2229] T 27226749529b464fb5097bc5677aedf1 P 9fae13b8fd8946ac82b9afee8651d5e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:30:45.135551 20514 raft_consensus.cc:2258] T 27226749529b464fb5097bc5677aedf1 P 9fae13b8fd8946ac82b9afee8651d5e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:30:45.141731 20514 ts_tablet_manager.cc:1905] T 27226749529b464fb5097bc5677aedf1 P 9fae13b8fd8946ac82b9afee8651d5e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:30:45.142204 20512 ts_tablet_manager.cc:1918] T 27226749529b464fb5097bc5677aedf1 P 8ccaf52c25044b2b97a6a34f34a8fe93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:30:45.142364 20512 log.cc:1196] T 27226749529b464fb5097bc5677aedf1 P 8ccaf52c25044b2b97a6a34f34a8fe93: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/27226749529b464fb5097bc5677aedf1
I20260928 08:30:45.142757 20513 ts_tablet_manager.cc:1918] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:30:45.142933 20513 log.cc:1196] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-0/wal/wals/27226749529b464fb5097bc5677aedf1
I20260928 08:30:45.142956 20512 ts_tablet_manager.cc:1939] T 27226749529b464fb5097bc5677aedf1 P 8ccaf52c25044b2b97a6a34f34a8fe93: Deleting consensus metadata
I20260928 08:30:45.143483 20513 ts_tablet_manager.cc:1939] T 27226749529b464fb5097bc5677aedf1 P c078080da9644522a981ed406e91bc03: Deleting consensus metadata
I20260928 08:30:45.144953 20000 catalog_manager.cc:4841] TS 8ccaf52c25044b2b97a6a34f34a8fe93 (127.15.134.194:35841): tablet 27226749529b464fb5097bc5677aedf1 (table test-table [id=1530838ae00040df9507060cc076c1c8]) successfully deleted
I20260928 08:30:45.145032 20001 catalog_manager.cc:4841] TS c078080da9644522a981ed406e91bc03 (127.15.134.193:36397): tablet 27226749529b464fb5097bc5677aedf1 (table test-table [id=1530838ae00040df9507060cc076c1c8]) successfully deleted
Sep 28 08:30:45 dist-test-slave-qp2n krb5kdc[19971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.193@KRBTEST.COM
I20260928 08:30:45.151206 20514 ts_tablet_manager.cc:1918] T 27226749529b464fb5097bc5677aedf1 P 9fae13b8fd8946ac82b9afee8651d5e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:30:45.151396 20514 log.cc:1196] T 27226749529b464fb5097bc5677aedf1 P 9fae13b8fd8946ac82b9afee8651d5e5: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/27226749529b464fb5097bc5677aedf1
I20260928 08:30:45.151943 20514 ts_tablet_manager.cc:1939] T 27226749529b464fb5097bc5677aedf1 P 9fae13b8fd8946ac82b9afee8651d5e5: Deleting consensus metadata
I20260928 08:30:45.153908 20002 catalog_manager.cc:4841] TS 9fae13b8fd8946ac82b9afee8651d5e5 (127.15.134.195:46147): tablet 27226749529b464fb5097bc5677aedf1 (table test-table [id=1530838ae00040df9507060cc076c1c8]) successfully deleted
I20260928 08:30:45.154804 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:45.123447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45750 (local address 127.15.134.193:36397)
0928 08:30:45.123632 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:45.123638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:45.124469 (+ 831us) server_negotiation.cc:407] Connection header received
0928 08:30:45.124717 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:45.124722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:45.124826 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:45.124948 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:45.136008 (+ 11060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:45.137368 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:45.139328 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:45.139992 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:45.140100 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:45.148276 (+ 8176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:45.148289 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:45.148304 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:45.148344 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:45.151705 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:45.152519 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:45.152527 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:45.152530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:45.152775 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:45.153156 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:45.153163 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:45.153166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:45.153505 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:45.153798 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:45.154177 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 08:30:45.154420 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 08:30:45.155474 20198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:45.257083 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 20076
I20260928 08:30:45.275998 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 20209
Sep 28 08:30:45 dist-test-slave-qp2n krb5kdc[19971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.194@KRBTEST.COM
I20260928 08:30:45.292254 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:45.276927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53180 (local address 127.15.134.194:35841)
0928 08:30:45.277098 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:45.277102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:45.277120 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:30:45.277179 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:45.277182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:45.277243 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:45.277346 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:45.278551 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:45.279311 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:45.280554 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:45.281141 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:45.281240 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:45.285650 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:45.285668 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:45.285686 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:45.285727 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:45.289269 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:45.289976 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:45.289984 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:45.289987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:45.290158 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:45.290525 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:45.290530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:45.290534 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:45.291002 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:45.291271 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:45.291694 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 08:30:45.291876 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 08:30:45.293001 20331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:45.382812 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 20209
I20260928 08:30:45.402241 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 20342
Sep 28 08:30:45 dist-test-slave-qp2n krb5kdc[19971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.195@KRBTEST.COM
I20260928 08:30:45.418964 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:45.403188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59010 (local address 127.15.134.195:46147)
0928 08:30:45.403355 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:45.403360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:45.403379 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:30:45.403506 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:45.403509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:45.403583 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:45.403679 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:45.405098 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:45.406011 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:45.407436 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:45.407953 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:45.408063 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:45.412690 (+ 4627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:45.412711 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:45.412729 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:45.412776 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:45.415485 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:45.416110 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:45.416117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:45.416119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:45.416259 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:45.416910 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:45.416914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:45.416916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:45.417376 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:45.417733 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:45.418138 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 08:30:45.418388 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 08:30:45.419762 20464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:45.521662 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 20342
I20260928 08:30:45.540534 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 19984
I20260928 08:30:45.541872 20045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:45.655505 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 19984
2026-09-28T08:30:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4380 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T08:30:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:30:45Z Disabled control of system clock
I20260928 08:30:45.725620 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:42819
--webserver_interface=127.15.134.254
--webserver_port=0
--builtin_ntp_servers=127.15.134.212:32791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:42819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 08:30:46.057971 20523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:46.058303 20523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:46.058390 20523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:46.065752 20523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:30:46.065865 20523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:46.065903 20523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:46.065951 20523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:30:46.065995 20523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:30:46.075461 20523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:32791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:42819
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:42819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:46.076813 20523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:46.078630 20523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:46.091079 20528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:46.091142 20531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:46.091109 20529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:46.092133 20523 server_base.cc:1034] running on GCE node
I20260928 08:30:46.092993 20523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:46.094525 20523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:46.095734 20523 hybrid_clock.cc:648] HybridClock initialized: now 1790584246095691 us; error 33 us; skew 500 ppm
I20260928 08:30:46.096110 20523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:30:46.098609 20523 webserver.cc:460] Webserver started at http://127.15.134.254:45573/ using document root <none> and password file <none>
I20260928 08:30:46.099346 20523 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:46.099589 20523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:46.100037 20523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:46.103165 20523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "f9c897cc3d9048309398264bbae77dd4"
format_stamp: "Formatted at 2026-09-28 08:30:46 on dist-test-slave-qp2n"
server_key: "d8840f7464fd4aa7c2db8517464031f9"
server_key_iv: "8fa7dc790356f717fde15caf7f72ae65"
server_key_version: "encryptionkey@0"
I20260928 08:30:46.103969 20523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "f9c897cc3d9048309398264bbae77dd4"
format_stamp: "Formatted at 2026-09-28 08:30:46 on dist-test-slave-qp2n"
server_key: "d8840f7464fd4aa7c2db8517464031f9"
server_key_iv: "8fa7dc790356f717fde15caf7f72ae65"
server_key_version: "encryptionkey@0"
I20260928 08:30:46.109496 20523 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260928 08:30:46.113070 20537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:46.114444 20523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 08:30:46.114576 20523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "f9c897cc3d9048309398264bbae77dd4"
format_stamp: "Formatted at 2026-09-28 08:30:46 on dist-test-slave-qp2n"
server_key: "d8840f7464fd4aa7c2db8517464031f9"
server_key_iv: "8fa7dc790356f717fde15caf7f72ae65"
server_key_version: "encryptionkey@0"
I20260928 08:30:46.114831 20523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:46.120710 20523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:46.121456 20523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:46.121779 20523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:46.140503 20523 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:42819
I20260928 08:30:46.140542 20588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:42819 every 8 connection(s)
I20260928 08:30:46.141950 20523 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:30:46.144332 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 20523
I20260928 08:30:46.144531 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:30:46.145252 15899 external_mini_cluster.cc:1428] Setting key f2ae255e4ed7608de8f1af3d6c6a1bd3
I20260928 08:30:46.145946 20589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:46.154227 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:46.147276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55436 (local address 127.15.134.254:42819)
0928 08:30:46.148272 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:46.148283 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:46.148323 (+ 40us) server_negotiation.cc:407] Connection header received
0928 08:30:46.150552 (+ 2229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:46.150573 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:46.151008 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:46.151408 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:46.152486 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:46.152492 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:46.152504 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:30:46.152522 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:46.152582 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:30:46.152649 (+ 67us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:46.152913 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:46.153060 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 08:30:46.153346 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":364,"threads_started":1}
I20260928 08:30:46.159081 20589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9c897cc3d9048309398264bbae77dd4: Bootstrap starting.
I20260928 08:30:46.162109 20589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9c897cc3d9048309398264bbae77dd4: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:46.163110 20589 log.cc:824] T 00000000000000000000000000000000 P f9c897cc3d9048309398264bbae77dd4: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:46.166117 20589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9c897cc3d9048309398264bbae77dd4: No bootstrap required, opened a new log
I20260928 08:30:46.173736 20589 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9c897cc3d9048309398264bbae77dd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c897cc3d9048309398264bbae77dd4" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 42819 } }
I20260928 08:30:46.174109 20589 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9c897cc3d9048309398264bbae77dd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:46.174182 20589 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9c897cc3d9048309398264bbae77dd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9c897cc3d9048309398264bbae77dd4, State: Initialized, Role: FOLLOWER
I20260928 08:30:46.174698 20589 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9c897cc3d9048309398264bbae77dd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c897cc3d9048309398264bbae77dd4" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 42819 } }
I20260928 08:30:46.174899 20589 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9c897cc3d9048309398264bbae77dd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:30:46.174998 20589 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9c897cc3d9048309398264bbae77dd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:30:46.175124 20589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9c897cc3d9048309398264bbae77dd4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:46.177434 20589 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9c897cc3d9048309398264bbae77dd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c897cc3d9048309398264bbae77dd4" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 42819 } }
I20260928 08:30:46.178045 20589 leader_election.cc:302] T 00000000000000000000000000000000 P f9c897cc3d9048309398264bbae77dd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9c897cc3d9048309398264bbae77dd4; no voters:
I20260928 08:30:46.178725 20589 leader_election.cc:288] T 00000000000000000000000000000000 P f9c897cc3d9048309398264bbae77dd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:30:46.178910 20594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9c897cc3d9048309398264bbae77dd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:46.179687 20594 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9c897cc3d9048309398264bbae77dd4 [term 1 LEADER]: Becoming Leader. State: Replica: f9c897cc3d9048309398264bbae77dd4, State: Running, Role: LEADER
I20260928 08:30:46.180240 20594 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9c897cc3d9048309398264bbae77dd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c897cc3d9048309398264bbae77dd4" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 42819 } }
I20260928 08:30:46.181056 20589 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9c897cc3d9048309398264bbae77dd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:30:46.183187 20595 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9c897cc3d9048309398264bbae77dd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9c897cc3d9048309398264bbae77dd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c897cc3d9048309398264bbae77dd4" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 42819 } } }
I20260928 08:30:46.183238 20596 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9c897cc3d9048309398264bbae77dd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9c897cc3d9048309398264bbae77dd4. Latest consensus state: current_term: 1 leader_uuid: "f9c897cc3d9048309398264bbae77dd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c897cc3d9048309398264bbae77dd4" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 42819 } } }
I20260928 08:30:46.183650 20595 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9c897cc3d9048309398264bbae77dd4 [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:46.183646 20596 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9c897cc3d9048309398264bbae77dd4 [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:46.185503 20601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:30:46.190841 20601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:30:46.200769 20601 catalog_manager.cc:1347] Generated new cluster ID: 5bd78a1b1072475eb860124eb2ff89c7
I20260928 08:30:46.200919 20601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:30:46.214934 20601 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:30:46.215427 20601 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:30:46.222476 20601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9c897cc3d9048309398264bbae77dd4: Generated new TSK 0
I20260928 08:30:46.223297 20601 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:30:46.239136 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:0
--local_ip_for_outbound_sockets=127.15.134.193
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:42819
--builtin_ntp_servers=127.15.134.212:32791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 08:30:46.592000 20613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:46.592293 20613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:46.592381 20613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:46.599871 20613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:46.599972 20613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:46.600090 20613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:30:46.609077 20613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:32791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:42819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:46.610478 20613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:46.612336 20613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:46.626156 20619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:46.626197 20621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:46.626240 20618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:46.630187 20613 server_base.cc:1034] running on GCE node
I20260928 08:30:46.630908 20613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:46.632277 20613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:46.633502 20613 hybrid_clock.cc:648] HybridClock initialized: now 1790584246633446 us; error 107 us; skew 500 ppm
I20260928 08:30:46.633911 20613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:30:46.636384 20613 webserver.cc:460] Webserver started at http://127.15.134.193:34799/ using document root <none> and password file <none>
I20260928 08:30:46.637269 20613 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:46.637360 20613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:46.637746 20613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:46.640739 20613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-0/data/instance:
uuid: "937c3712327a4cd6b582bcd2ed77b1e4"
format_stamp: "Formatted at 2026-09-28 08:30:46 on dist-test-slave-qp2n"
server_key: "7fcdab9dd50afd0505062af2727e9f8b"
server_key_iv: "ec5498cb8f58f840bc02939ae8625266"
server_key_version: "encryptionkey@0"
I20260928 08:30:46.641552 20613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance:
uuid: "937c3712327a4cd6b582bcd2ed77b1e4"
format_stamp: "Formatted at 2026-09-28 08:30:46 on dist-test-slave-qp2n"
server_key: "7fcdab9dd50afd0505062af2727e9f8b"
server_key_iv: "ec5498cb8f58f840bc02939ae8625266"
server_key_version: "encryptionkey@0"
I20260928 08:30:46.647027 20613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:30:46.650555 20627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:46.651991 20613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:30:46.652177 20613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "937c3712327a4cd6b582bcd2ed77b1e4"
format_stamp: "Formatted at 2026-09-28 08:30:46 on dist-test-slave-qp2n"
server_key: "7fcdab9dd50afd0505062af2727e9f8b"
server_key_iv: "ec5498cb8f58f840bc02939ae8625266"
server_key_version: "encryptionkey@0"
I20260928 08:30:46.652407 20613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:46.658185 20613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:46.659052 20613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:46.659333 20613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:46.660269 20613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:30:46.661805 20613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:46.661931 20613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:46.662070 20613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:46.662160 20613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:46.696571 20613 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:46263
I20260928 08:30:46.696799 20739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:46263 every 8 connection(s)
I20260928 08:30:46.698236 20613 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:30:46.702044 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 20613
I20260928 08:30:46.702256 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance
I20260928 08:30:46.703234 15899 external_mini_cluster.cc:1428] Setting key 55e781b7ff20d72f2f2c00d85854b5a1
I20260928 08:30:46.706573 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:0
--local_ip_for_outbound_sockets=127.15.134.194
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:42819
--builtin_ntp_servers=127.15.134.212:32791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 08:30:46.708552 20742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:46.701264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:37019 (local address 127.15.134.254:42819)
0928 08:30:46.704160 (+ 2896us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:46.704167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:46.704190 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:30:46.706208 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:46.706212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:46.706261 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:46.706349 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:46.707879 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:46.707883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:46.707886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:30:46.707902 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:46.707943 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:30:46.707955 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:46.708053 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:46.708286 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 08:30:46.708335 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2772,"thread_start_us":2666,"threads_started":1}
I20260928 08:30:46.709903 20743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:46.702158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:42819 (local address 127.15.134.193:37019)
0928 08:30:46.703794 (+ 1636us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:46.703846 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:46.705899 (+ 2053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:46.706609 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:46.706624 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:46.706760 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:46.707510 (+ 750us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:30:46.707516 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:46.707573 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:30:46.707698 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:46.708063 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:46.708075 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:46.708085 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 08:30:46.708364 (+ 279us) client_negotiation.cc:240] Negotiation successful
0928 08:30:46.708660 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"thread_start_us":501,"threads_started":1}
I20260928 08:30:46.711700 20740 heartbeater.cc:347] Connected to a master server at 127.15.134.254:42819
I20260928 08:30:46.712004 20740 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:46.712690 20740 heartbeater.cc:511] Master 127.15.134.254:42819 requested a full tablet report, sending...
I20260928 08:30:46.715176 20554 ts_manager.cc:194] Registered new tserver with Master: 937c3712327a4cd6b582bcd2ed77b1e4 (127.15.134.193:46263)
W20260928 08:30:47.038878 20744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:47.039196 20744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:47.039346 20744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:47.047216 20744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:47.047327 20744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:47.047497 20744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:30:47.057588 20744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:32791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:42819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:47.059027 20744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:47.061035 20744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:47.075312 20749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:47.075282 20750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:47.075335 20752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:47.076453 20744 server_base.cc:1034] running on GCE node
I20260928 08:30:47.077243 20744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:47.078277 20744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:47.079488 20744 hybrid_clock.cc:648] HybridClock initialized: now 1790584247079390 us; error 103 us; skew 500 ppm
I20260928 08:30:47.079840 20744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:30:47.082532 20744 webserver.cc:460] Webserver started at http://127.15.134.194:38973/ using document root <none> and password file <none>
I20260928 08:30:47.083290 20744 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:47.083410 20744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:47.083770 20744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:47.086983 20744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-1/data/instance:
uuid: "727afbcaa3a840bb97609ff9a4d3eaa3"
format_stamp: "Formatted at 2026-09-28 08:30:47 on dist-test-slave-qp2n"
server_key: "f50f0172830ea9b837671d209446b314"
server_key_iv: "7cf6166dd58d203596577bc6f3c4e5df"
server_key_version: "encryptionkey@0"
I20260928 08:30:47.087831 20744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance:
uuid: "727afbcaa3a840bb97609ff9a4d3eaa3"
format_stamp: "Formatted at 2026-09-28 08:30:47 on dist-test-slave-qp2n"
server_key: "f50f0172830ea9b837671d209446b314"
server_key_iv: "7cf6166dd58d203596577bc6f3c4e5df"
server_key_version: "encryptionkey@0"
I20260928 08:30:47.093183 20744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260928 08:30:47.096915 20758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:47.098425 20744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:30:47.098634 20744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "727afbcaa3a840bb97609ff9a4d3eaa3"
format_stamp: "Formatted at 2026-09-28 08:30:47 on dist-test-slave-qp2n"
server_key: "f50f0172830ea9b837671d209446b314"
server_key_iv: "7cf6166dd58d203596577bc6f3c4e5df"
server_key_version: "encryptionkey@0"
I20260928 08:30:47.098873 20744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:47.104954 20744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:47.105784 20744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:47.106065 20744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:47.106974 20744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:30:47.108379 20744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:47.108497 20744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:47.108649 20744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:47.108747 20744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260928 08:30:47.142295 20744 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:35615
I20260928 08:30:47.142326 20870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:35615 every 8 connection(s)
I20260928 08:30:47.143846 20744 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:30:47.149433 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 20744
I20260928 08:30:47.149646 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance
I20260928 08:30:47.150235 15899 external_mini_cluster.cc:1428] Setting key df252b58a92483921d4d370abe6c993e
I20260928 08:30:47.153307 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:0
--local_ip_for_outbound_sockets=127.15.134.195
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:42819
--builtin_ntp_servers=127.15.134.212:32791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 08:30:47.153594 20742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:47.146668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:55859 (local address 127.15.134.254:42819)
0928 08:30:47.146849 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:47.146854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:47.148488 (+ 1634us) server_negotiation.cc:407] Connection header received
0928 08:30:47.150982 (+ 2494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:47.150988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:47.151098 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:47.151220 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:47.152713 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:47.152719 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:47.152723 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:30:47.152743 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:47.152782 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:30:47.152797 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:47.152924 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:47.153189 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 08:30:47.153258 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 08:30:47.154645 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:47.147334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:42819 (local address 127.15.134.194:55859)
0928 08:30:47.148320 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:47.148357 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:47.150661 (+ 2304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:47.151453 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:47.151461 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:47.151558 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:47.152324 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:30:47.152328 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:47.152380 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:30:47.152501 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:47.152911 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:47.152921 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:47.152929 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 08:30:47.153241 (+ 312us) client_negotiation.cc:240] Negotiation successful
0928 08:30:47.153495 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":333,"threads_started":1}
I20260928 08:30:47.156225 20871 heartbeater.cc:347] Connected to a master server at 127.15.134.254:42819
I20260928 08:30:47.156522 20871 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:47.157356 20871 heartbeater.cc:511] Master 127.15.134.254:42819 requested a full tablet report, sending...
I20260928 08:30:47.158948 20554 ts_manager.cc:194] Registered new tserver with Master: 727afbcaa3a840bb97609ff9a4d3eaa3 (127.15.134.194:35615)
W20260928 08:30:47.503079 20874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:47.503369 20874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:47.503552 20874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:47.511047 20874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:47.511163 20874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:47.511338 20874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:30:47.521270 20874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:32791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:42819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:47.522557 20874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:47.524326 20874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:47.538053 20879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:47.538259 20882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:47.538110 20880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:47.542387 20874 server_base.cc:1034] running on GCE node
I20260928 08:30:47.543044 20874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:47.544039 20874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:47.545279 20874 hybrid_clock.cc:648] HybridClock initialized: now 1790584247545242 us; error 49 us; skew 500 ppm
I20260928 08:30:47.545644 20874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:30:47.547994 20874 webserver.cc:460] Webserver started at http://127.15.134.195:45641/ using document root <none> and password file <none>
I20260928 08:30:47.548944 20874 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:47.549074 20874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:47.549397 20874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:47.552325 20874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-2/data/instance:
uuid: "8b31b36321004a449b978bd777761c0c"
format_stamp: "Formatted at 2026-09-28 08:30:47 on dist-test-slave-qp2n"
server_key: "2cd738d8333a17b6010460ac3ec1d589"
server_key_iv: "a65c9d361d305c17bc87b8a27099c7ff"
server_key_version: "encryptionkey@0"
I20260928 08:30:47.553186 20874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance:
uuid: "8b31b36321004a449b978bd777761c0c"
format_stamp: "Formatted at 2026-09-28 08:30:47 on dist-test-slave-qp2n"
server_key: "2cd738d8333a17b6010460ac3ec1d589"
server_key_iv: "a65c9d361d305c17bc87b8a27099c7ff"
server_key_version: "encryptionkey@0"
I20260928 08:30:47.558916 20874 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 08:30:47.562914 20888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:47.564431 20874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:30:47.564682 20874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "8b31b36321004a449b978bd777761c0c"
format_stamp: "Formatted at 2026-09-28 08:30:47 on dist-test-slave-qp2n"
server_key: "2cd738d8333a17b6010460ac3ec1d589"
server_key_iv: "a65c9d361d305c17bc87b8a27099c7ff"
server_key_version: "encryptionkey@0"
I20260928 08:30:47.564982 20874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:47.571756 20874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:47.572597 20874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:47.572921 20874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:47.573865 20874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:30:47.575385 20874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:47.575523 20874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:47.575650 20874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:47.575743 20874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:47.610741 20874 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:39521
I20260928 08:30:47.610831 21000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:39521 every 8 connection(s)
I20260928 08:30:47.612069 20874 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:30:47.616211 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 20874
I20260928 08:30:47.616391 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance
I20260928 08:30:47.617010 15899 external_mini_cluster.cc:1428] Setting key 06fd12f219103d9c2b2e4a8614ebffa3
I20260928 08:30:47.621528 20742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:47.614932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:52911 (local address 127.15.134.254:42819)
0928 08:30:47.615169 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:47.615175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:47.616825 (+ 1650us) server_negotiation.cc:407] Connection header received
0928 08:30:47.619029 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:47.619032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:47.619123 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:47.619264 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:47.620756 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:47.620764 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:47.620768 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:30:47.620786 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:47.620828 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:30:47.620843 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:47.620946 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:47.621155 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 08:30:47.621221 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:30:47.622427 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:47.615555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:42819 (local address 127.15.134.195:52911)
0928 08:30:47.616660 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:47.616714 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:47.618698 (+ 1984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:47.619503 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:47.619511 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:47.619615 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:47.620358 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:30:47.620362 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:47.620417 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:30:47.620544 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:47.620922 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:47.620932 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:47.620940 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 08:30:47.621238 (+ 298us) client_negotiation.cc:240] Negotiation successful
0928 08:30:47.621487 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":330,"threads_started":1}
I20260928 08:30:47.623721 21001 heartbeater.cc:347] Connected to a master server at 127.15.134.254:42819
I20260928 08:30:47.624002 21001 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:47.624571 21001 heartbeater.cc:511] Master 127.15.134.254:42819 requested a full tablet report, sending...
I20260928 08:30:47.625941 20554 ts_manager.cc:194] Registered new tserver with Master: 8b31b36321004a449b978bd777761c0c (127.15.134.195:39521)
I20260928 08:30:47.632954 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:30:47.640120 20742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:47.637837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51540 (local address 127.15.134.254:42819)
0928 08:30:47.638074 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:47.638079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:47.638335 (+ 256us) server_negotiation.cc:407] Connection header received
0928 08:30:47.638478 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:47.638482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:47.638552 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:47.638658 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:47.639561 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:47.639565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:47.639568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:30:47.639584 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:47.639615 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:30:47.639627 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:47.639731 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:47.639835 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 08:30:47.639883 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 08:30:47.644183 20554 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:30:47.651314 20554 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:30:47.684725 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:47.665481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54436 (local address 127.15.134.194:35615)
0928 08:30:47.681614 (+ 16133us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:47.681624 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:47.681660 (+ 36us) server_negotiation.cc:407] Connection header received
0928 08:30:47.681738 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:47.681748 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:47.682016 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:47.682174 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:47.683695 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:47.683705 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:47.683720 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:30:47.683753 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:47.683801 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:30:47.683816 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:47.683969 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:47.684214 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 08:30:47.684323 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15964,"thread_start_us":15838,"threads_started":1}
I20260928 08:30:47.684814 21013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:47.665303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:35615 (local address 127.0.0.1:54436)
0928 08:30:47.680733 (+ 15430us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:47.680798 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:47.681013 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:47.682394 (+ 1381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:47.682401 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:47.682496 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:47.683421 (+ 925us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:30:47.683424 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:47.683449 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:30:47.683516 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:47.683924 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:47.683936 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:47.683943 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 08:30:47.684232 (+ 289us) client_negotiation.cc:240] Negotiation successful
0928 08:30:47.684331 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14998,"thread_start_us":16268,"threads_started":1}
I20260928 08:30:47.688794 20806 tablet_service.cc:1467] Processing CreateTablet for tablet d61e4d7b9fb745b3b10bed1d7abd077e (DEFAULT_TABLE table=test-table [id=ab5113d19e534403bb5630d1498552d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:47.690045 20806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d61e4d7b9fb745b3b10bed1d7abd077e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:47.694813 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:47.666575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43614 (local address 127.15.134.195:39521)
0928 08:30:47.690428 (+ 23853us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:47.690441 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:47.691365 (+ 924us) server_negotiation.cc:407] Connection header received
0928 08:30:47.691521 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:47.691529 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:47.691765 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:47.691939 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:47.693829 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:47.693839 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:47.693854 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:30:47.693873 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:47.693930 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:30:47.693952 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:47.694133 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:47.694214 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 08:30:47.694313 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23582,"thread_start_us":23877,"threads_started":1}
I20260928 08:30:47.695276 21016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:47.665970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:39521 (local address 127.0.0.1:43614)
0928 08:30:47.691192 (+ 25222us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:47.691218 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:47.691370 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:47.692714 (+ 1344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:47.692719 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:47.692748 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:47.693579 (+ 831us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:30:47.693582 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:47.693599 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:30:47.693647 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:47.694073 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:47.694079 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:47.694085 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 08:30:47.694230 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 08:30:47.694302 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25031,"thread_start_us":25534,"threads_started":1}
I20260928 08:30:47.696154 21017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:47.666390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42794 (local address 127.15.134.193:46263)
0928 08:30:47.692563 (+ 26173us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:47.692573 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:47.692604 (+ 31us) server_negotiation.cc:407] Connection header received
0928 08:30:47.692702 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:47.692710 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:47.692924 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:47.693745 (+ 821us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:47.694762 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:47.694770 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:47.694781 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:30:47.694808 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:47.694861 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:30:47.694881 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:47.695028 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:47.695654 (+ 626us) server_negotiation.cc:299] Negotiation successful
0928 08:30:47.695754 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25981,"thread_start_us":26241,"threads_started":1}
I20260928 08:30:47.695591 21013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:47.665764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:46263 (local address 127.0.0.1:42794)
0928 08:30:47.685049 (+ 19285us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:47.685079 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:47.685168 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:47.693121 (+ 7953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:47.693128 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:47.693169 (+ 41us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:47.693756 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:30:47.693761 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:47.693781 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:30:47.693833 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:47.695092 (+ 1259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:47.695103 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:47.695108 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:30:47.695232 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 08:30:47.695299 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19189,"thread_start_us":26185,"threads_started":1}
I20260928 08:30:47.700677 20936 tablet_service.cc:1467] Processing CreateTablet for tablet d61e4d7b9fb745b3b10bed1d7abd077e (DEFAULT_TABLE table=test-table [id=ab5113d19e534403bb5630d1498552d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:47.701475 20675 tablet_service.cc:1467] Processing CreateTablet for tablet d61e4d7b9fb745b3b10bed1d7abd077e (DEFAULT_TABLE table=test-table [id=ab5113d19e534403bb5630d1498552d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:30:47.702790 20936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d61e4d7b9fb745b3b10bed1d7abd077e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:47.703052 20675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d61e4d7b9fb745b3b10bed1d7abd077e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:47.707017 21019 tablet_bootstrap.cc:492] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3: Bootstrap starting.
I20260928 08:30:47.711417 21019 tablet_bootstrap.cc:654] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:47.712893 21019 log.cc:824] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:47.716768 21019 tablet_bootstrap.cc:492] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3: No bootstrap required, opened a new log
I20260928 08:30:47.717178 21019 ts_tablet_manager.cc:1397] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s
I20260928 08:30:47.717216 21021 tablet_bootstrap.cc:492] T d61e4d7b9fb745b3b10bed1d7abd077e P 8b31b36321004a449b978bd777761c0c: Bootstrap starting.
I20260928 08:30:47.718417 20740 heartbeater.cc:503] Master 127.15.134.254:42819 was elected leader, sending a full tablet report...
I20260928 08:30:47.722494 21021 tablet_bootstrap.cc:654] T d61e4d7b9fb745b3b10bed1d7abd077e P 8b31b36321004a449b978bd777761c0c: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:47.723031 21022 tablet_bootstrap.cc:492] T d61e4d7b9fb745b3b10bed1d7abd077e P 937c3712327a4cd6b582bcd2ed77b1e4: Bootstrap starting.
I20260928 08:30:47.723817 21021 log.cc:824] T d61e4d7b9fb745b3b10bed1d7abd077e P 8b31b36321004a449b978bd777761c0c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:47.726547 21022 tablet_bootstrap.cc:654] T d61e4d7b9fb745b3b10bed1d7abd077e P 937c3712327a4cd6b582bcd2ed77b1e4: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:47.727253 21021 tablet_bootstrap.cc:492] T d61e4d7b9fb745b3b10bed1d7abd077e P 8b31b36321004a449b978bd777761c0c: No bootstrap required, opened a new log
I20260928 08:30:47.727535 21021 ts_tablet_manager.cc:1397] T d61e4d7b9fb745b3b10bed1d7abd077e P 8b31b36321004a449b978bd777761c0c: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260928 08:30:47.727730 21022 log.cc:824] T d61e4d7b9fb745b3b10bed1d7abd077e P 937c3712327a4cd6b582bcd2ed77b1e4: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:47.729590 21019 raft_consensus.cc:348] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727afbcaa3a840bb97609ff9a4d3eaa3" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35615 } } peers { permanent_uuid: "937c3712327a4cd6b582bcd2ed77b1e4" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 46263 } } peers { permanent_uuid: "8b31b36321004a449b978bd777761c0c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 39521 } }
I20260928 08:30:47.730101 21019 raft_consensus.cc:374] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:47.730226 21019 raft_consensus.cc:729] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 727afbcaa3a840bb97609ff9a4d3eaa3, State: Initialized, Role: FOLLOWER
I20260928 08:30:47.731024 21022 tablet_bootstrap.cc:492] T d61e4d7b9fb745b3b10bed1d7abd077e P 937c3712327a4cd6b582bcd2ed77b1e4: No bootstrap required, opened a new log
I20260928 08:30:47.730945 21019 consensus_queue.cc:260] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727afbcaa3a840bb97609ff9a4d3eaa3" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35615 } } peers { permanent_uuid: "937c3712327a4cd6b582bcd2ed77b1e4" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 46263 } } peers { permanent_uuid: "8b31b36321004a449b978bd777761c0c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 39521 } }
I20260928 08:30:47.731349 21022 ts_tablet_manager.cc:1397] T d61e4d7b9fb745b3b10bed1d7abd077e P 937c3712327a4cd6b582bcd2ed77b1e4: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 08:30:47.733317 20871 heartbeater.cc:503] Master 127.15.134.254:42819 was elected leader, sending a full tablet report...
I20260928 08:30:47.735587 21019 ts_tablet_manager.cc:1428] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s
I20260928 08:30:47.737366 21021 raft_consensus.cc:348] T d61e4d7b9fb745b3b10bed1d7abd077e P 8b31b36321004a449b978bd777761c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727afbcaa3a840bb97609ff9a4d3eaa3" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35615 } } peers { permanent_uuid: "937c3712327a4cd6b582bcd2ed77b1e4" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 46263 } } peers { permanent_uuid: "8b31b36321004a449b978bd777761c0c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 39521 } }
I20260928 08:30:47.737847 21021 raft_consensus.cc:374] T d61e4d7b9fb745b3b10bed1d7abd077e P 8b31b36321004a449b978bd777761c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:47.738004 21021 raft_consensus.cc:729] T d61e4d7b9fb745b3b10bed1d7abd077e P 8b31b36321004a449b978bd777761c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b31b36321004a449b978bd777761c0c, State: Initialized, Role: FOLLOWER
I20260928 08:30:47.738749 21021 consensus_queue.cc:260] T d61e4d7b9fb745b3b10bed1d7abd077e P 8b31b36321004a449b978bd777761c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727afbcaa3a840bb97609ff9a4d3eaa3" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35615 } } peers { permanent_uuid: "937c3712327a4cd6b582bcd2ed77b1e4" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 46263 } } peers { permanent_uuid: "8b31b36321004a449b978bd777761c0c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 39521 } }
I20260928 08:30:47.739850 21001 heartbeater.cc:503] Master 127.15.134.254:42819 was elected leader, sending a full tablet report...
I20260928 08:30:47.740499 21021 ts_tablet_manager.cc:1428] T d61e4d7b9fb745b3b10bed1d7abd077e P 8b31b36321004a449b978bd777761c0c: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260928 08:30:47.741508 21022 raft_consensus.cc:348] T d61e4d7b9fb745b3b10bed1d7abd077e P 937c3712327a4cd6b582bcd2ed77b1e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727afbcaa3a840bb97609ff9a4d3eaa3" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35615 } } peers { permanent_uuid: "937c3712327a4cd6b582bcd2ed77b1e4" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 46263 } } peers { permanent_uuid: "8b31b36321004a449b978bd777761c0c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 39521 } }
I20260928 08:30:47.741899 21022 raft_consensus.cc:374] T d61e4d7b9fb745b3b10bed1d7abd077e P 937c3712327a4cd6b582bcd2ed77b1e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:47.742022 21022 raft_consensus.cc:729] T d61e4d7b9fb745b3b10bed1d7abd077e P 937c3712327a4cd6b582bcd2ed77b1e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 937c3712327a4cd6b582bcd2ed77b1e4, State: Initialized, Role: FOLLOWER
I20260928 08:30:47.742698 21022 consensus_queue.cc:260] T d61e4d7b9fb745b3b10bed1d7abd077e P 937c3712327a4cd6b582bcd2ed77b1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727afbcaa3a840bb97609ff9a4d3eaa3" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35615 } } peers { permanent_uuid: "937c3712327a4cd6b582bcd2ed77b1e4" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 46263 } } peers { permanent_uuid: "8b31b36321004a449b978bd777761c0c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 39521 } }
I20260928 08:30:47.744674 21022 ts_tablet_manager.cc:1428] T d61e4d7b9fb745b3b10bed1d7abd077e P 937c3712327a4cd6b582bcd2ed77b1e4: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260928 08:30:47.863963 21002 tablet.cc:2369] T d61e4d7b9fb745b3b10bed1d7abd077e P 8b31b36321004a449b978bd777761c0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:30:47.896101 20872 tablet.cc:2369] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:30:47.939198 21025 raft_consensus.cc:482] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:30:47.939553 21025 raft_consensus.cc:504] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727afbcaa3a840bb97609ff9a4d3eaa3" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35615 } } peers { permanent_uuid: "937c3712327a4cd6b582bcd2ed77b1e4" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 46263 } } peers { permanent_uuid: "8b31b36321004a449b978bd777761c0c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 39521 } }
I20260928 08:30:47.941444 21025 leader_election.cc:288] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 937c3712327a4cd6b582bcd2ed77b1e4 (127.15.134.193:46263), 8b31b36321004a449b978bd777761c0c (127.15.134.195:39521)
I20260928 08:30:47.945333 21029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:47.941960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:39521 (local address 127.15.134.194:33681)
0928 08:30:47.942817 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:47.942833 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:47.942998 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:47.943376 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:47.943381 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:47.943406 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:47.944196 (+ 790us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:30:47.944198 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:47.944211 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:30:47.944249 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:47.944646 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:47.944651 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:47.944656 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:30:47.944814 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 08:30:47.944884 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":617,"threads_started":1}
I20260928 08:30:47.945333 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:47.942044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:33681 (local address 127.15.134.195:39521)
0928 08:30:47.942266 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:47.942274 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:47.942957 (+ 683us) server_negotiation.cc:407] Connection header received
0928 08:30:47.943137 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:47.943146 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:47.943221 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:47.943306 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:47.944395 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:47.944401 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:47.944406 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:30:47.944425 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:47.944457 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:30:47.944471 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:47.944667 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:47.944940 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 08:30:47.945019 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 08:30:47.945472 21017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:47.941896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:35853 (local address 127.15.134.193:46263)
0928 08:30:47.942124 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:47.942133 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:47.943016 (+ 883us) server_negotiation.cc:407] Connection header received
0928 08:30:47.943271 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:47.943307 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:47.943372 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:47.943481 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:47.944648 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:47.944653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:47.944657 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:30:47.944677 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:47.944718 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:30:47.944738 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:47.944851 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:47.945119 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 08:30:47.945184 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 08:30:47.945745 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:47.941678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:46263 (local address 127.15.134.194:35853)
0928 08:30:47.942832 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:47.942857 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:47.943099 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:47.943520 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:47.943525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:47.943550 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:47.944394 (+ 844us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:30:47.944398 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:47.944413 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:30:47.944456 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:47.944913 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:47.944919 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:47.944930 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 08:30:47.945149 (+ 219us) client_negotiation.cc:240] Negotiation successful
0928 08:30:47.945209 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"spinlock_wait_cycles":53504,"thread_start_us":657,"threads_started":1}
I20260928 08:30:47.946791 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61e4d7b9fb745b3b10bed1d7abd077e" candidate_uuid: "727afbcaa3a840bb97609ff9a4d3eaa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b31b36321004a449b978bd777761c0c" is_pre_election: true
I20260928 08:30:47.946796 20695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61e4d7b9fb745b3b10bed1d7abd077e" candidate_uuid: "727afbcaa3a840bb97609ff9a4d3eaa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "937c3712327a4cd6b582bcd2ed77b1e4" is_pre_election: true
I20260928 08:30:47.947324 20956 raft_consensus.cc:2454] T d61e4d7b9fb745b3b10bed1d7abd077e P 8b31b36321004a449b978bd777761c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 727afbcaa3a840bb97609ff9a4d3eaa3 in term 0.
I20260928 08:30:47.947353 20695 raft_consensus.cc:2454] T d61e4d7b9fb745b3b10bed1d7abd077e P 937c3712327a4cd6b582bcd2ed77b1e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 727afbcaa3a840bb97609ff9a4d3eaa3 in term 0.
I20260928 08:30:47.948100 20759 leader_election.cc:302] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 727afbcaa3a840bb97609ff9a4d3eaa3, 8b31b36321004a449b978bd777761c0c, 937c3712327a4cd6b582bcd2ed77b1e4; no voters:
I20260928 08:30:47.948537 21025 raft_consensus.cc:2781] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:30:47.948762 21025 raft_consensus.cc:482] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:30:47.948896 21025 raft_consensus.cc:3037] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3 [term 0 FOLLOWER]: Advancing to term 1
W20260928 08:30:47.950838 20741 tablet.cc:2369] T d61e4d7b9fb745b3b10bed1d7abd077e P 937c3712327a4cd6b582bcd2ed77b1e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:30:47.951404 21025 raft_consensus.cc:504] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727afbcaa3a840bb97609ff9a4d3eaa3" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35615 } } peers { permanent_uuid: "937c3712327a4cd6b582bcd2ed77b1e4" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 46263 } } peers { permanent_uuid: "8b31b36321004a449b978bd777761c0c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 39521 } }
I20260928 08:30:47.952121 21025 leader_election.cc:288] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3 [CANDIDATE]: Term 1 election: Requested vote from peers 937c3712327a4cd6b582bcd2ed77b1e4 (127.15.134.193:46263), 8b31b36321004a449b978bd777761c0c (127.15.134.195:39521)
I20260928 08:30:47.952705 20956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61e4d7b9fb745b3b10bed1d7abd077e" candidate_uuid: "727afbcaa3a840bb97609ff9a4d3eaa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b31b36321004a449b978bd777761c0c"
I20260928 08:30:47.952705 20695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d61e4d7b9fb745b3b10bed1d7abd077e" candidate_uuid: "727afbcaa3a840bb97609ff9a4d3eaa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "937c3712327a4cd6b582bcd2ed77b1e4"
I20260928 08:30:47.952946 20956 raft_consensus.cc:3037] T d61e4d7b9fb745b3b10bed1d7abd077e P 8b31b36321004a449b978bd777761c0c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:47.952994 20695 raft_consensus.cc:3037] T d61e4d7b9fb745b3b10bed1d7abd077e P 937c3712327a4cd6b582bcd2ed77b1e4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:47.956478 20695 raft_consensus.cc:2454] T d61e4d7b9fb745b3b10bed1d7abd077e P 937c3712327a4cd6b582bcd2ed77b1e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 727afbcaa3a840bb97609ff9a4d3eaa3 in term 1.
I20260928 08:30:47.956486 20956 raft_consensus.cc:2454] T d61e4d7b9fb745b3b10bed1d7abd077e P 8b31b36321004a449b978bd777761c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 727afbcaa3a840bb97609ff9a4d3eaa3 in term 1.
I20260928 08:30:47.957048 20760 leader_election.cc:302] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 727afbcaa3a840bb97609ff9a4d3eaa3, 8b31b36321004a449b978bd777761c0c; no voters:
I20260928 08:30:47.957399 21025 raft_consensus.cc:2781] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:47.957873 21025 raft_consensus.cc:686] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3 [term 1 LEADER]: Becoming Leader. State: Replica: 727afbcaa3a840bb97609ff9a4d3eaa3, State: Running, Role: LEADER
I20260928 08:30:47.958354 21025 consensus_queue.cc:237] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727afbcaa3a840bb97609ff9a4d3eaa3" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35615 } } peers { permanent_uuid: "937c3712327a4cd6b582bcd2ed77b1e4" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 46263 } } peers { permanent_uuid: "8b31b36321004a449b978bd777761c0c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 39521 } }
I20260928 08:30:47.963231 20553 catalog_manager.cc:5495] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 727afbcaa3a840bb97609ff9a4d3eaa3 (127.15.134.194). New cstate: current_term: 1 leader_uuid: "727afbcaa3a840bb97609ff9a4d3eaa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727afbcaa3a840bb97609ff9a4d3eaa3" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "937c3712327a4cd6b582bcd2ed77b1e4" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 46263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b31b36321004a449b978bd777761c0c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 39521 } health_report { overall_health: UNKNOWN } } }
I20260928 08:30:48.002456 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:48.000308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54438 (local address 127.15.134.194:35615)
0928 08:30:48.000508 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:48.000513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:48.000532 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:30:48.000738 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:48.000742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:48.000845 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:48.000938 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:48.001820 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:48.001825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:48.001828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:30:48.001843 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:48.001873 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:30:48.001884 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:48.001978 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:48.002076 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 08:30:48.002132 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 08:30:48.010937 20956 raft_consensus.cc:1261] T d61e4d7b9fb745b3b10bed1d7abd077e P 8b31b36321004a449b978bd777761c0c [term 1 FOLLOWER]: Refusing update from remote peer 727afbcaa3a840bb97609ff9a4d3eaa3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:30:48.010949 20695 raft_consensus.cc:1261] T d61e4d7b9fb745b3b10bed1d7abd077e P 937c3712327a4cd6b582bcd2ed77b1e4 [term 1 FOLLOWER]: Refusing update from remote peer 727afbcaa3a840bb97609ff9a4d3eaa3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:30:48.011881 21030 consensus_queue.cc:1035] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b31b36321004a449b978bd777761c0c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 39521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:30:48.012197 21025 consensus_queue.cc:1035] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "937c3712327a4cd6b582bcd2ed77b1e4" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 46263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:30:48.041051 20553 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:30:48.041373 20553 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:30:48.044948 20553 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9c897cc3d9048309398264bbae77dd4: Sending DeleteTablet for 3 replicas of tablet d61e4d7b9fb745b3b10bed1d7abd077e
I20260928 08:30:48.046132 20806 tablet_service.cc:1514] Processing DeleteTablet for tablet d61e4d7b9fb745b3b10bed1d7abd077e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:30:48 UTC) from {username='slave'} at 127.0.0.1:54436
I20260928 08:30:48.046335 20936 tablet_service.cc:1514] Processing DeleteTablet for tablet d61e4d7b9fb745b3b10bed1d7abd077e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:30:48 UTC) from {username='slave'} at 127.0.0.1:43614
I20260928 08:30:48.046386 20675 tablet_service.cc:1514] Processing DeleteTablet for tablet d61e4d7b9fb745b3b10bed1d7abd077e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:30:48 UTC) from {username='slave'} at 127.0.0.1:42794
I20260928 08:30:48.050820 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 20613
I20260928 08:30:48.058754 21043 tablet_replica.cc:330] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3: stopping tablet replica
I20260928 08:30:48.059381 21043 raft_consensus.cc:2229] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:30:48.060220 21043 raft_consensus.cc:2258] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:30:48.062623 21044 tablet_replica.cc:330] T d61e4d7b9fb745b3b10bed1d7abd077e P 8b31b36321004a449b978bd777761c0c: stopping tablet replica
I20260928 08:30:48.063031 21045 tablet_replica.cc:330] T d61e4d7b9fb745b3b10bed1d7abd077e P 937c3712327a4cd6b582bcd2ed77b1e4: stopping tablet replica
I20260928 08:30:48.063457 21043 ts_tablet_manager.cc:1905] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:30:48.063526 21017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:48.052455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42798 (local address 127.15.134.193:46263)
0928 08:30:48.052708 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:48.052717 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:48.060764 (+ 8047us) server_negotiation.cc:407] Connection header received
0928 08:30:48.061043 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:48.061053 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:48.061221 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:48.061318 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:48.062708 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:48.062714 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:48.062718 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:30:48.062741 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:48.062781 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:30:48.062797 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:48.062912 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:48.063174 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 08:30:48.063236 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 08:30:48.064215 21045 raft_consensus.cc:2229] T d61e4d7b9fb745b3b10bed1d7abd077e P 937c3712327a4cd6b582bcd2ed77b1e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:30:48.064769 21045 raft_consensus.cc:2258] T d61e4d7b9fb745b3b10bed1d7abd077e P 937c3712327a4cd6b582bcd2ed77b1e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:30:48.063565 21044 raft_consensus.cc:2229] T d61e4d7b9fb745b3b10bed1d7abd077e P 8b31b36321004a449b978bd777761c0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:30:48.065501 21044 raft_consensus.cc:2258] T d61e4d7b9fb745b3b10bed1d7abd077e P 8b31b36321004a449b978bd777761c0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:30:48.065346 20735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:48.069804 21043 ts_tablet_manager.cc:1918] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:30:48.070026 21043 log.cc:1196] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/d61e4d7b9fb745b3b10bed1d7abd077e
I20260928 08:30:48.070640 21043 ts_tablet_manager.cc:1939] T d61e4d7b9fb745b3b10bed1d7abd077e P 727afbcaa3a840bb97609ff9a4d3eaa3: Deleting consensus metadata
I20260928 08:30:48.070788 21044 ts_tablet_manager.cc:1905] T d61e4d7b9fb745b3b10bed1d7abd077e P 8b31b36321004a449b978bd777761c0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:30:48.072666 20541 catalog_manager.cc:4841] TS 727afbcaa3a840bb97609ff9a4d3eaa3 (127.15.134.194:35615): tablet d61e4d7b9fb745b3b10bed1d7abd077e (table test-table [id=ab5113d19e534403bb5630d1498552d5]) successfully deleted
I20260928 08:30:48.076407 21044 ts_tablet_manager.cc:1918] T d61e4d7b9fb745b3b10bed1d7abd077e P 8b31b36321004a449b978bd777761c0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:30:48.076593 21044 log.cc:1196] T d61e4d7b9fb745b3b10bed1d7abd077e P 8b31b36321004a449b978bd777761c0c: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/d61e4d7b9fb745b3b10bed1d7abd077e
I20260928 08:30:48.077147 21044 ts_tablet_manager.cc:1939] T d61e4d7b9fb745b3b10bed1d7abd077e P 8b31b36321004a449b978bd777761c0c: Deleting consensus metadata
I20260928 08:30:48.079129 20539 catalog_manager.cc:4841] TS 8b31b36321004a449b978bd777761c0c (127.15.134.195:39521): tablet d61e4d7b9fb745b3b10bed1d7abd077e (table test-table [id=ab5113d19e534403bb5630d1498552d5]) successfully deleted
I20260928 08:30:48.175958 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 20613
I20260928 08:30:48.199868 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 20744
W20260928 08:30:48.199916 20538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.134.193:46263 (error 108)
I20260928 08:30:48.202271 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:48.201271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:46263 (local address 127.0.0.1:42802)
0928 08:30:48.201769 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:48.201941 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:46263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":278,"threads_started":1}
W20260928 08:30:48.202512 20538 catalog_manager.cc:4584] TS 937c3712327a4cd6b582bcd2ed77b1e4 (127.15.134.193:46263): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d61e4d7b9fb745b3b10bed1d7abd077e: Network error: Client connection negotiation failed: client connection to 127.15.134.193:46263: connect: Connection refused (error 111)
I20260928 08:30:48.203845 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:48.201011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54444 (local address 127.15.134.194:35615)
0928 08:30:48.201221 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:48.201228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:48.201734 (+ 506us) server_negotiation.cc:407] Connection header received
0928 08:30:48.201834 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:48.201839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:48.201912 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:48.202024 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:48.203075 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:48.203080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:48.203084 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:30:48.203100 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:48.203139 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:30:48.203177 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:48.203303 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:48.203495 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 08:30:48.203565 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 08:30:48.204852 20866 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:48.256356 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:48.255748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:46263 (local address 127.0.0.1:42810)
0928 08:30:48.256033 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:48.256179 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:46263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 08:30:48.257509 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:48.257086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:46263 (local address 127.0.0.1:42820)
0928 08:30:48.257262 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:48.257349 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:46263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 08:30:48.314888 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 20744
I20260928 08:30:48.317998 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:48.317464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:46263 (local address 127.0.0.1:42836)
0928 08:30:48.317663 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:48.317759 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:46263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 08:30:48.319078 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:48.318638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:46263 (local address 127.0.0.1:42844)
0928 08:30:48.318807 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:48.318887 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:46263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 08:30:48.333925 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 20874
I20260928 08:30:48.336510 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:48.334673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43620 (local address 127.15.134.195:39521)
0928 08:30:48.334830 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:48.334834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:48.334867 (+ 33us) server_negotiation.cc:407] Connection header received
0928 08:30:48.335018 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:48.335022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:48.335122 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:48.335208 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:48.335997 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:48.336001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:48.336004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:30:48.336019 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:48.336047 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:30:48.336072 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:48.336175 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:48.336246 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 08:30:48.336290 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 08:30:48.337231 20996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:48.399837 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:48.399236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:46263 (local address 127.0.0.1:42858)
0928 08:30:48.399512 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:48.399636 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:46263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260928 08:30:48.401181 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:48.400719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:46263 (local address 127.0.0.1:42870)
0928 08:30:48.400926 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:48.401014 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:46263: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260928 08:30:48.429296 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 20874
I20260928 08:30:48.446766 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 20523
I20260928 08:30:48.448137 20584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:48.558708 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 20523
2026-09-28T08:30:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2902 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 08:30:48.602842 15899 mini_oidc.cc:152] Starting JWKS server
I20260928 08:30:48.605645 15899 webserver.cc:460] Webserver started at https://127.0.0.1:39161/ using document root <none> and password file <none>
I20260928 08:30:48.605999 15899 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 08:30:48.606760 15899 webserver.cc:460] Webserver started at http://127.0.0.1:42733/ using document root <none> and password file <none>
2026-09-28T08:30:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:30:48Z Disabled control of system clock
I20260928 08:30:48.636270 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:45891
--webserver_interface=127.15.134.254
--webserver_port=0
--builtin_ntp_servers=127.15.134.212:41853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:45891
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39161/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584205252592-15899-0/testchainca.cert with env {}
W20260928 08:30:48.975162 21058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:48.975574 21058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:48.975732 21058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:48.983006 21058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:30:48.983186 21058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:48.983280 21058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:48.983390 21058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:30:48.983482 21058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 08:30:48.983579 21058 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 08:30:48.983640 21058 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39161/jwks.json
I20260928 08:30:48.994238 21058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:41853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:45891
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:45891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39161/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584205252592-15899-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:48.995700 21058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:48.997193 21058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:49.010041 21064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:49.010126 21063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:49.010165 21066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:49.011408 21058 server_base.cc:1034] running on GCE node
I20260928 08:30:49.012234 21058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:49.013763 21058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:49.014984 21058 hybrid_clock.cc:648] HybridClock initialized: now 1790584249014912 us; error 54 us; skew 500 ppm
I20260928 08:30:49.015514 21058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:30:49.018486 21058 webserver.cc:460] Webserver started at http://127.15.134.254:44523/ using document root <none> and password file <none>
I20260928 08:30:49.019310 21058 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:49.019452 21058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:49.019824 21058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:49.023113 21058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "80fe3bafe2054590a86be070600a7796"
format_stamp: "Formatted at 2026-09-28 08:30:49 on dist-test-slave-qp2n"
server_key: "3c9fc2d3484a444e763ad2ff4077b17e"
server_key_iv: "9048e23efb1fccbd50135ab6f32d8eaf"
server_key_version: "encryptionkey@0"
I20260928 08:30:49.024003 21058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "80fe3bafe2054590a86be070600a7796"
format_stamp: "Formatted at 2026-09-28 08:30:49 on dist-test-slave-qp2n"
server_key: "3c9fc2d3484a444e763ad2ff4077b17e"
server_key_iv: "9048e23efb1fccbd50135ab6f32d8eaf"
server_key_version: "encryptionkey@0"
I20260928 08:30:49.029717 21058 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 08:30:49.033638 21072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:49.035018 21058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:30:49.035231 21058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "80fe3bafe2054590a86be070600a7796"
format_stamp: "Formatted at 2026-09-28 08:30:49 on dist-test-slave-qp2n"
server_key: "3c9fc2d3484a444e763ad2ff4077b17e"
server_key_iv: "9048e23efb1fccbd50135ab6f32d8eaf"
server_key_version: "encryptionkey@0"
I20260928 08:30:49.035463 21058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:49.051965 21058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:49.053066 21058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:49.053431 21058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:49.073385 21058 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:45891
I20260928 08:30:49.073407 21123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:45891 every 8 connection(s)
I20260928 08:30:49.074940 21058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:30:49.076915 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 21058
I20260928 08:30:49.077243 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:30:49.077982 15899 external_mini_cluster.cc:1428] Setting key 16b5e8f962606e645c10f8d56a5d9b54
I20260928 08:30:49.079562 21124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:49.093590 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:49.080164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36266 (local address 127.15.134.254:45891)
0928 08:30:49.081557 (+ 1393us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:49.081569 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:49.081619 (+ 50us) server_negotiation.cc:407] Connection header received
0928 08:30:49.083484 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:49.083508 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:49.084009 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:49.084372 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:49.085129 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:49.086665 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:49.088732 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:49.089426 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:49.089575 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:49.090981 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:49.090989 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:49.091007 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:30:49.091665 (+ 658us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:49.091750 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:30:49.091771 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:49.091870 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:49.092112 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 08:30:49.092557 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":460,"threads_started":1}
I20260928 08:30:49.094396 21124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80fe3bafe2054590a86be070600a7796: Bootstrap starting.
I20260928 08:30:49.097791 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:30:49.098058 21124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80fe3bafe2054590a86be070600a7796: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:49.099042 21124 log.cc:824] T 00000000000000000000000000000000 P 80fe3bafe2054590a86be070600a7796: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:49.102331 21124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80fe3bafe2054590a86be070600a7796: No bootstrap required, opened a new log
I20260928 08:30:49.109741 21124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80fe3bafe2054590a86be070600a7796 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fe3bafe2054590a86be070600a7796" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45891 } }
I20260928 08:30:49.110134 21124 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80fe3bafe2054590a86be070600a7796 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:49.110242 21124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80fe3bafe2054590a86be070600a7796 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fe3bafe2054590a86be070600a7796, State: Initialized, Role: FOLLOWER
I20260928 08:30:49.110857 21124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80fe3bafe2054590a86be070600a7796 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fe3bafe2054590a86be070600a7796" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45891 } }
I20260928 08:30:49.111105 21124 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80fe3bafe2054590a86be070600a7796 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:30:49.111250 21124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80fe3bafe2054590a86be070600a7796 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:30:49.111436 21124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80fe3bafe2054590a86be070600a7796 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:49.113628 21124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80fe3bafe2054590a86be070600a7796 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fe3bafe2054590a86be070600a7796" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45891 } }
I20260928 08:30:49.114127 21124 leader_election.cc:302] T 00000000000000000000000000000000 P 80fe3bafe2054590a86be070600a7796 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80fe3bafe2054590a86be070600a7796; no voters:
I20260928 08:30:49.114845 21124 leader_election.cc:288] T 00000000000000000000000000000000 P 80fe3bafe2054590a86be070600a7796 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:30:49.115156 21129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80fe3bafe2054590a86be070600a7796 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:49.115901 21129 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80fe3bafe2054590a86be070600a7796 [term 1 LEADER]: Becoming Leader. State: Replica: 80fe3bafe2054590a86be070600a7796, State: Running, Role: LEADER
I20260928 08:30:49.116441 21129 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80fe3bafe2054590a86be070600a7796 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fe3bafe2054590a86be070600a7796" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45891 } }
I20260928 08:30:49.117101 21124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80fe3bafe2054590a86be070600a7796 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:30:49.119482 21131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80fe3bafe2054590a86be070600a7796 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80fe3bafe2054590a86be070600a7796. Latest consensus state: current_term: 1 leader_uuid: "80fe3bafe2054590a86be070600a7796" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fe3bafe2054590a86be070600a7796" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45891 } } }
I20260928 08:30:49.119551 21130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80fe3bafe2054590a86be070600a7796 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80fe3bafe2054590a86be070600a7796" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80fe3bafe2054590a86be070600a7796" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45891 } } }
I20260928 08:30:49.119874 21131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80fe3bafe2054590a86be070600a7796 [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:49.119931 21130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80fe3bafe2054590a86be070600a7796 [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:49.122031 21136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:30:49.126776 21136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:30:49.136125 21136 catalog_manager.cc:1347] Generated new cluster ID: 3fe15118ccc84d2da7addc5083e88f7a
I20260928 08:30:49.136240 21136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:30:49.158830 21136 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:30:49.160964 21136 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:30:49.170169 21136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80fe3bafe2054590a86be070600a7796: Generated new TSK 0
I20260928 08:30:49.171187 21136 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 08:30:49.221414 21154 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 08:30:49.221769 21154 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 08:30:49.222561 21127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:49.194065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36280 (local address 127.15.134.254:45891)
0928 08:30:49.194254 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:49.194262 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:49.194781 (+ 519us) server_negotiation.cc:407] Connection header received
0928 08:30:49.195128 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:49.195133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:49.195218 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:49.195361 (+ 143us) server_negotiation.cc:226] Negotiated authn=JWT
0928 08:30:49.195916 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:49.196749 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:49.198181 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:49.198655 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:49.198797 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:49.199321 (+ 524us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 08:30:49.221806 (+ 22485us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39161/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 08:30:49.222235 (+ 429us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36280: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39161/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 08:30:49.222707 21127 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36280: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39161/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 08:30:49.224301 21153 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:30:49.194007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:45891 (local address 127.0.0.1:36280)
0928 08:30:49.194672 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:49.194689 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:49.194887 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:49.195396 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:49.195401 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:49.195415 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0928 08:30:49.195757 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:49.195765 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:49.196887 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:49.196897 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:49.198030 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:49.198041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:49.198842 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:49.198849 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:49.198986 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:49.199046 (+ 60us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 08:30:49.222174 (+ 23128us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39161/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 08:30:49.222542 (+ 368us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.134.254:45891: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39161/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":340,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.15.134.254:45891: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39161/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 08:30:49.226157 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 21058
I20260928 08:30:49.227376 21119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:49.322935 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 21058
2026-09-28T08:30:49Z chronyd exiting
I20260928 08:30:49.374920 15899 test_util.cc:175] -----------------------------------------------
I20260928 08:30:49.375110 15899 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790584205252592-15899-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (777 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 08:30:49.378966 15899 mini_oidc.cc:152] Starting JWKS server
I20260928 08:30:49.380898 15899 webserver.cc:460] Webserver started at https://127.0.0.1:33059/ using document root <none> and password file <none>
I20260928 08:30:49.381145 15899 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 08:30:49.381827 15899 webserver.cc:460] Webserver started at http://127.0.0.1:42455/ using document root <none> and password file <none>
2026-09-28T08:30:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:30:49Z Disabled control of system clock
I20260928 08:30:49.410396 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:39111
--webserver_interface=127.15.134.254
--webserver_port=0
--builtin_ntp_servers=127.15.134.212:39237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:39111
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33059/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584205252592-15899-0/testchainca.cert with env {}
W20260928 08:30:49.741109 21161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:49.741412 21161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:49.741509 21161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:49.749042 21161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:30:49.749155 21161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:49.749192 21161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:49.749241 21161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:30:49.749285 21161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 08:30:49.749341 21161 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 08:30:49.749382 21161 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33059/jwks.json
I20260928 08:30:49.759670 21161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:39237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:39111
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:39111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33059/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584205252592-15899-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:49.761049 21161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:49.762315 21161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:49.774955 21166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:49.774998 21169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:49.774998 21167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:49.775908 21161 server_base.cc:1034] running on GCE node
I20260928 08:30:49.776836 21161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:49.778479 21161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:49.779722 21161 hybrid_clock.cc:648] HybridClock initialized: now 1790584249779647 us; error 71 us; skew 500 ppm
I20260928 08:30:49.780100 21161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:30:49.787751 21161 webserver.cc:460] Webserver started at http://127.15.134.254:36185/ using document root <none> and password file <none>
I20260928 08:30:49.788585 21161 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:49.788789 21161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:49.789144 21161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:49.792189 21161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "db6cbf1ababa4f20a38b4dee3283a59b"
format_stamp: "Formatted at 2026-09-28 08:30:49 on dist-test-slave-qp2n"
server_key: "080aa75b2819167d7ee839e4d085cf92"
server_key_iv: "4b3db97974295f0f2d866de45c234c41"
server_key_version: "encryptionkey@0"
I20260928 08:30:49.793075 21161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "db6cbf1ababa4f20a38b4dee3283a59b"
format_stamp: "Formatted at 2026-09-28 08:30:49 on dist-test-slave-qp2n"
server_key: "080aa75b2819167d7ee839e4d085cf92"
server_key_iv: "4b3db97974295f0f2d866de45c234c41"
server_key_version: "encryptionkey@0"
I20260928 08:30:49.798545 21161 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:30:49.802487 21175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:49.804240 21161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:30:49.804488 21161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "db6cbf1ababa4f20a38b4dee3283a59b"
format_stamp: "Formatted at 2026-09-28 08:30:49 on dist-test-slave-qp2n"
server_key: "080aa75b2819167d7ee839e4d085cf92"
server_key_iv: "4b3db97974295f0f2d866de45c234c41"
server_key_version: "encryptionkey@0"
I20260928 08:30:49.804801 21161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:49.827210 21161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:49.828117 21161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:49.828460 21161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:49.847983 21161 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:39111
I20260928 08:30:49.847991 21226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:39111 every 8 connection(s)
I20260928 08:30:49.849455 21161 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:30:49.850768 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 21161
I20260928 08:30:49.850992 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:30:49.851732 15899 external_mini_cluster.cc:1428] Setting key 22208d7102333c5754c213cefaafe5b8
I20260928 08:30:49.853915 21227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:49.867882 21230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:49.853788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34960 (local address 127.15.134.254:39111)
0928 08:30:49.854918 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:49.854932 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:49.854984 (+ 52us) server_negotiation.cc:407] Connection header received
0928 08:30:49.857116 (+ 2132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:49.857143 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:49.857771 (+ 628us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:49.858224 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:49.858769 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:49.860866 (+ 2097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:49.862307 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:49.863331 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:49.863528 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:49.864799 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:49.864809 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:49.864835 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:30:49.865645 (+ 810us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:49.865739 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:30:49.865771 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:49.865901 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:49.866134 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 08:30:49.866515 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":385,"threads_started":1}
I20260928 08:30:49.869045 21227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db6cbf1ababa4f20a38b4dee3283a59b: Bootstrap starting.
I20260928 08:30:49.872354 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:30:49.872958 21227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db6cbf1ababa4f20a38b4dee3283a59b: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:49.874176 21227 log.cc:824] T 00000000000000000000000000000000 P db6cbf1ababa4f20a38b4dee3283a59b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:49.877405 21227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db6cbf1ababa4f20a38b4dee3283a59b: No bootstrap required, opened a new log
I20260928 08:30:49.885072 21227 raft_consensus.cc:348] T 00000000000000000000000000000000 P db6cbf1ababa4f20a38b4dee3283a59b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6cbf1ababa4f20a38b4dee3283a59b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 39111 } }
I20260928 08:30:49.885470 21227 raft_consensus.cc:374] T 00000000000000000000000000000000 P db6cbf1ababa4f20a38b4dee3283a59b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:49.885551 21227 raft_consensus.cc:729] T 00000000000000000000000000000000 P db6cbf1ababa4f20a38b4dee3283a59b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db6cbf1ababa4f20a38b4dee3283a59b, State: Initialized, Role: FOLLOWER
I20260928 08:30:49.886065 21227 consensus_queue.cc:260] T 00000000000000000000000000000000 P db6cbf1ababa4f20a38b4dee3283a59b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6cbf1ababa4f20a38b4dee3283a59b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 39111 } }
I20260928 08:30:49.886307 21227 raft_consensus.cc:388] T 00000000000000000000000000000000 P db6cbf1ababa4f20a38b4dee3283a59b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:30:49.886451 21227 raft_consensus.cc:482] T 00000000000000000000000000000000 P db6cbf1ababa4f20a38b4dee3283a59b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:30:49.886641 21227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db6cbf1ababa4f20a38b4dee3283a59b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:49.889073 21227 raft_consensus.cc:504] T 00000000000000000000000000000000 P db6cbf1ababa4f20a38b4dee3283a59b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6cbf1ababa4f20a38b4dee3283a59b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 39111 } }
I20260928 08:30:49.889587 21227 leader_election.cc:302] T 00000000000000000000000000000000 P db6cbf1ababa4f20a38b4dee3283a59b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db6cbf1ababa4f20a38b4dee3283a59b; no voters:
I20260928 08:30:49.890192 21227 leader_election.cc:288] T 00000000000000000000000000000000 P db6cbf1ababa4f20a38b4dee3283a59b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:30:49.890357 21232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db6cbf1ababa4f20a38b4dee3283a59b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:49.891197 21232 raft_consensus.cc:686] T 00000000000000000000000000000000 P db6cbf1ababa4f20a38b4dee3283a59b [term 1 LEADER]: Becoming Leader. State: Replica: db6cbf1ababa4f20a38b4dee3283a59b, State: Running, Role: LEADER
I20260928 08:30:49.891685 21232 consensus_queue.cc:237] T 00000000000000000000000000000000 P db6cbf1ababa4f20a38b4dee3283a59b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6cbf1ababa4f20a38b4dee3283a59b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 39111 } }
I20260928 08:30:49.892349 21227 sys_catalog.cc:564] T 00000000000000000000000000000000 P db6cbf1ababa4f20a38b4dee3283a59b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:30:49.894516 21234 sys_catalog.cc:455] T 00000000000000000000000000000000 P db6cbf1ababa4f20a38b4dee3283a59b [sys.catalog]: SysCatalogTable state changed. Reason: New leader db6cbf1ababa4f20a38b4dee3283a59b. Latest consensus state: current_term: 1 leader_uuid: "db6cbf1ababa4f20a38b4dee3283a59b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6cbf1ababa4f20a38b4dee3283a59b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 39111 } } }
I20260928 08:30:49.894953 21234 sys_catalog.cc:458] T 00000000000000000000000000000000 P db6cbf1ababa4f20a38b4dee3283a59b [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:49.894924 21233 sys_catalog.cc:455] T 00000000000000000000000000000000 P db6cbf1ababa4f20a38b4dee3283a59b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db6cbf1ababa4f20a38b4dee3283a59b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db6cbf1ababa4f20a38b4dee3283a59b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 39111 } } }
I20260928 08:30:49.895234 21233 sys_catalog.cc:458] T 00000000000000000000000000000000 P db6cbf1ababa4f20a38b4dee3283a59b [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:49.895948 21240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:30:49.902230 21240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:30:49.910926 21240 catalog_manager.cc:1347] Generated new cluster ID: 721e158819b34acfb7769759fc7b9221
I20260928 08:30:49.911070 21240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:30:49.922837 21240 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:30:49.924280 21240 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:30:49.933756 21240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db6cbf1ababa4f20a38b4dee3283a59b: Generated new TSK 0
I20260928 08:30:49.934684 21240 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 08:30:49.971784 21257 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 08:30:49.971915 21257 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 08:30:49.973241 21230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:49.952819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34972 (local address 127.15.134.254:39111)
0928 08:30:49.953030 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:49.953036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:49.953636 (+ 600us) server_negotiation.cc:407] Connection header received
0928 08:30:49.953840 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:49.953848 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:49.953948 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:49.954145 (+ 197us) server_negotiation.cc:226] Negotiated authn=JWT
0928 08:30:49.954634 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:49.955778 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:49.957217 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:49.957909 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:49.958046 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:49.958491 (+ 445us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 08:30:49.972475 (+ 13984us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 08:30:49.972886 (+ 411us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34972: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 08:30:49.973258 21256 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:30:49.952813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39111 (local address 127.0.0.1:34972)
0928 08:30:49.953523 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:49.953541 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:49.953673 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:49.954175 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:49.954178 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:49.954189 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0928 08:30:49.954424 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:49.954431 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:49.955925 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:49.955928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:49.957062 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:49.957074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:49.958077 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:49.958082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:49.958206 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:49.958238 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 08:30:49.972753 (+ 14515us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 08:30:49.972955 (+ 202us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.134.254:39111: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":393,"threads_started":1}
W20260928 08:30:49.973363 21230 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34972: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33059/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260928 08:30:49.974529 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 21161
I20260928 08:30:49.976253 21222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:50.069527 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 21161
2026-09-28T08:30:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (747 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 08:30:50.126029 15899 mini_oidc.cc:152] Starting JWKS server
I20260928 08:30:50.127882 15899 webserver.cc:460] Webserver started at https://127.0.0.1:38917/ using document root <none> and password file <none>
I20260928 08:30:50.128160 15899 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 08:30:50.128912 15899 webserver.cc:460] Webserver started at http://127.0.0.1:37141/ using document root <none> and password file <none>
2026-09-28T08:30:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:30:50Z Disabled control of system clock
I20260928 08:30:50.156013 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:41577
--webserver_interface=127.15.134.254
--webserver_port=0
--builtin_ntp_servers=127.15.134.212:33017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:41577
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38917/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 08:30:50.486886 21264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:50.487301 21264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:50.487474 21264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:50.494886 21264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:30:50.495059 21264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:50.495158 21264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:50.495270 21264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:30:50.495357 21264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 08:30:50.495446 21264 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 08:30:50.495537 21264 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38917/jwks.json
I20260928 08:30:50.504889 21264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:33017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:41577
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:41577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38917/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:50.506260 21264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:50.508121 21264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:50.521498 21270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:50.521790 21269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:50.521536 21272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:50.522837 21264 server_base.cc:1034] running on GCE node
I20260928 08:30:50.523743 21264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:50.525262 21264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:50.526557 21264 hybrid_clock.cc:648] HybridClock initialized: now 1790584250526523 us; error 48 us; skew 500 ppm
I20260928 08:30:50.526916 21264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:30:50.529315 21264 webserver.cc:460] Webserver started at http://127.15.134.254:46339/ using document root <none> and password file <none>
I20260928 08:30:50.530050 21264 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:50.530194 21264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:50.530541 21264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:50.533558 21264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "a880fd7fc4984112a63cf68722e2c6ea"
format_stamp: "Formatted at 2026-09-28 08:30:50 on dist-test-slave-qp2n"
server_key: "cbd597c543c067a4c5b7ca060cf277a4"
server_key_iv: "77f49e7f09b76b11a5cf0c588ed58b83"
server_key_version: "encryptionkey@0"
I20260928 08:30:50.534421 21264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "a880fd7fc4984112a63cf68722e2c6ea"
format_stamp: "Formatted at 2026-09-28 08:30:50 on dist-test-slave-qp2n"
server_key: "cbd597c543c067a4c5b7ca060cf277a4"
server_key_iv: "77f49e7f09b76b11a5cf0c588ed58b83"
server_key_version: "encryptionkey@0"
I20260928 08:30:50.540194 21264 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 08:30:50.544198 21278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:50.545914 21264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 08:30:50.546125 21264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "a880fd7fc4984112a63cf68722e2c6ea"
format_stamp: "Formatted at 2026-09-28 08:30:50 on dist-test-slave-qp2n"
server_key: "cbd597c543c067a4c5b7ca060cf277a4"
server_key_iv: "77f49e7f09b76b11a5cf0c588ed58b83"
server_key_version: "encryptionkey@0"
I20260928 08:30:50.546368 21264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:50.559271 21264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:50.560232 21264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:50.560588 21264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:50.580557 21264 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:41577
I20260928 08:30:50.580565 21329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:41577 every 8 connection(s)
I20260928 08:30:50.582017 21264 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:30:50.586035 21330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:50.586570 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 21264
I20260928 08:30:50.586814 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:30:50.587531 15899 external_mini_cluster.cc:1428] Setting key e1ffbdef69ea4d8eef9de02c26d85d8e
I20260928 08:30:50.600169 21330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a880fd7fc4984112a63cf68722e2c6ea: Bootstrap starting.
I20260928 08:30:50.602237 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:50.589723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35794 (local address 127.15.134.254:41577)
0928 08:30:50.590881 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:50.590892 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:50.590948 (+ 56us) server_negotiation.cc:407] Connection header received
0928 08:30:50.592583 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:50.592605 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:50.593157 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:50.593503 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:50.594087 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:50.595480 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:50.597435 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:50.598133 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:50.598260 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:50.599588 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:50.599597 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:50.599612 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:30:50.600133 (+ 521us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:50.600215 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:30:50.600236 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:50.600484 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:50.600810 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 08:30:50.601124 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":405,"threads_started":1}
I20260928 08:30:50.603803 21330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a880fd7fc4984112a63cf68722e2c6ea: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:50.604959 21330 log.cc:824] T 00000000000000000000000000000000 P a880fd7fc4984112a63cf68722e2c6ea: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:50.606482 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:30:50.608207 21330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a880fd7fc4984112a63cf68722e2c6ea: No bootstrap required, opened a new log
I20260928 08:30:50.616322 21330 raft_consensus.cc:348] T 00000000000000000000000000000000 P a880fd7fc4984112a63cf68722e2c6ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a880fd7fc4984112a63cf68722e2c6ea" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41577 } }
I20260928 08:30:50.616781 21330 raft_consensus.cc:374] T 00000000000000000000000000000000 P a880fd7fc4984112a63cf68722e2c6ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:50.616907 21330 raft_consensus.cc:729] T 00000000000000000000000000000000 P a880fd7fc4984112a63cf68722e2c6ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a880fd7fc4984112a63cf68722e2c6ea, State: Initialized, Role: FOLLOWER
I20260928 08:30:50.617532 21330 consensus_queue.cc:260] T 00000000000000000000000000000000 P a880fd7fc4984112a63cf68722e2c6ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a880fd7fc4984112a63cf68722e2c6ea" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41577 } }
I20260928 08:30:50.617830 21330 raft_consensus.cc:388] T 00000000000000000000000000000000 P a880fd7fc4984112a63cf68722e2c6ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:30:50.618018 21330 raft_consensus.cc:482] T 00000000000000000000000000000000 P a880fd7fc4984112a63cf68722e2c6ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:30:50.618219 21330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a880fd7fc4984112a63cf68722e2c6ea [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:50.620533 21330 raft_consensus.cc:504] T 00000000000000000000000000000000 P a880fd7fc4984112a63cf68722e2c6ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a880fd7fc4984112a63cf68722e2c6ea" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41577 } }
I20260928 08:30:50.621141 21330 leader_election.cc:302] T 00000000000000000000000000000000 P a880fd7fc4984112a63cf68722e2c6ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a880fd7fc4984112a63cf68722e2c6ea; no voters:
I20260928 08:30:50.621771 21330 leader_election.cc:288] T 00000000000000000000000000000000 P a880fd7fc4984112a63cf68722e2c6ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:30:50.621882 21335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a880fd7fc4984112a63cf68722e2c6ea [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:50.622859 21335 raft_consensus.cc:686] T 00000000000000000000000000000000 P a880fd7fc4984112a63cf68722e2c6ea [term 1 LEADER]: Becoming Leader. State: Replica: a880fd7fc4984112a63cf68722e2c6ea, State: Running, Role: LEADER
I20260928 08:30:50.623399 21335 consensus_queue.cc:237] T 00000000000000000000000000000000 P a880fd7fc4984112a63cf68722e2c6ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a880fd7fc4984112a63cf68722e2c6ea" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41577 } }
I20260928 08:30:50.624135 21330 sys_catalog.cc:564] T 00000000000000000000000000000000 P a880fd7fc4984112a63cf68722e2c6ea [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:30:50.626561 21336 sys_catalog.cc:455] T 00000000000000000000000000000000 P a880fd7fc4984112a63cf68722e2c6ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a880fd7fc4984112a63cf68722e2c6ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a880fd7fc4984112a63cf68722e2c6ea" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41577 } } }
I20260928 08:30:50.626919 21336 sys_catalog.cc:458] T 00000000000000000000000000000000 P a880fd7fc4984112a63cf68722e2c6ea [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:50.626652 21337 sys_catalog.cc:455] T 00000000000000000000000000000000 P a880fd7fc4984112a63cf68722e2c6ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader a880fd7fc4984112a63cf68722e2c6ea. Latest consensus state: current_term: 1 leader_uuid: "a880fd7fc4984112a63cf68722e2c6ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a880fd7fc4984112a63cf68722e2c6ea" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41577 } } }
I20260928 08:30:50.627087 21337 sys_catalog.cc:458] T 00000000000000000000000000000000 P a880fd7fc4984112a63cf68722e2c6ea [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:50.629311 21340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:30:50.634781 21340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:30:50.643949 21340 catalog_manager.cc:1347] Generated new cluster ID: b8c4ac25d1b54d20902ee5ce6deec642
I20260928 08:30:50.644074 21340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:30:50.674384 21340 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:30:50.675848 21340 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:30:50.685168 21340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a880fd7fc4984112a63cf68722e2c6ea: Generated new TSK 0
I20260928 08:30:50.686074 21340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:30:50.714248 21333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:50.705633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35808 (local address 127.15.134.254:41577)
0928 08:30:50.705878 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:50.705888 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:50.706388 (+ 500us) server_negotiation.cc:407] Connection header received
0928 08:30:50.706667 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:50.706672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:50.706746 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:50.706882 (+ 136us) server_negotiation.cc:226] Negotiated authn=JWT
0928 08:30:50.707446 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:50.708233 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:50.710369 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:50.710859 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:50.710957 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:50.711600 (+ 643us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 08:30:50.713569 (+ 1969us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38917/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 08:30:50.713905 (+ 336us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35808: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38917/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":77}
W20260928 08:30:50.714485 21359 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:30:50.705589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41577 (local address 127.0.0.1:35808)
0928 08:30:50.706263 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:50.706284 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:50.706440 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:50.706933 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:50.706938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:50.706953 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0928 08:30:50.707288 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:50.707300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:50.708378 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:50.708384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:50.710212 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:50.710228 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:50.711057 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:50.711064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:50.711253 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:50.711304 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 08:30:50.713774 (+ 2470us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38917/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 08:30:50.714048 (+ 274us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.134.254:41577: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38917/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":364,"threads_started":1}
W20260928 08:30:50.714783 21333 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35808: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38917/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260928 08:30:50.715361 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 21264
I20260928 08:30:50.716507 21325 generic_service.cc:196] Checking for leaks (request via RPC)
E20260928 08:30:50.726922 21360 webserver.cc:533] Webserver: sslize failed:
E20260928 08:30:50.727048 21360 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 08:30:50.810797 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 21264
2026-09-28T08:30:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (741 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T08:30:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:30:50Z Disabled control of system clock
I20260928 08:30:50.897886 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:33629
--webserver_interface=127.15.134.254
--webserver_port=0
--builtin_ntp_servers=127.15.134.212:37135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:33629
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584205252592-15899-0/testchainca.cert with env {}
W20260928 08:30:51.234193 21365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:51.234515 21365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:51.234683 21365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:51.242292 21365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:30:51.242412 21365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:51.242534 21365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:51.242641 21365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:30:51.242719 21365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 08:30:51.242810 21365 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 08:30:51.242864 21365 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 08:30:51.253485 21365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:37135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:33629
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:33629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584205252592-15899-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:51.254878 21365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:51.256243 21365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:51.268736 21370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:51.268844 21373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:51.268997 21371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:51.269651 21365 server_base.cc:1034] running on GCE node
I20260928 08:30:51.270382 21365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:51.271771 21365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:51.272969 21365 hybrid_clock.cc:648] HybridClock initialized: now 1790584251272926 us; error 34 us; skew 500 ppm
I20260928 08:30:51.273288 21365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:30:51.275903 21365 webserver.cc:460] Webserver started at http://127.15.134.254:41457/ using document root <none> and password file <none>
I20260928 08:30:51.276664 21365 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:51.276777 21365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:51.277045 21365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:51.279924 21365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "d0ad74794d8c4019b4179ae77bc11a14"
format_stamp: "Formatted at 2026-09-28 08:30:51 on dist-test-slave-qp2n"
server_key: "d5b907443eb05d4f5b825f489a7007f4"
server_key_iv: "89547ac5ce370c4fb572054ded06a1bd"
server_key_version: "encryptionkey@0"
I20260928 08:30:51.280764 21365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "d0ad74794d8c4019b4179ae77bc11a14"
format_stamp: "Formatted at 2026-09-28 08:30:51 on dist-test-slave-qp2n"
server_key: "d5b907443eb05d4f5b825f489a7007f4"
server_key_iv: "89547ac5ce370c4fb572054ded06a1bd"
server_key_version: "encryptionkey@0"
I20260928 08:30:51.286047 21365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 08:30:51.290057 21379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:51.291538 21365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:30:51.291741 21365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "d0ad74794d8c4019b4179ae77bc11a14"
format_stamp: "Formatted at 2026-09-28 08:30:51 on dist-test-slave-qp2n"
server_key: "d5b907443eb05d4f5b825f489a7007f4"
server_key_iv: "89547ac5ce370c4fb572054ded06a1bd"
server_key_version: "encryptionkey@0"
I20260928 08:30:51.291970 21365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:51.307471 21365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:51.308339 21365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:51.308678 21365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:51.327708 21365 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:33629
I20260928 08:30:51.327729 21430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:33629 every 8 connection(s)
I20260928 08:30:51.329160 21365 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:30:51.333243 21431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:51.339165 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 21365
I20260928 08:30:51.339365 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:30:51.340260 15899 external_mini_cluster.cc:1428] Setting key ff932d6e149a776571a87562b05a2dde
I20260928 08:30:51.345804 21431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0ad74794d8c4019b4179ae77bc11a14: Bootstrap starting.
I20260928 08:30:51.349750 21431 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0ad74794d8c4019b4179ae77bc11a14: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:51.351166 21431 log.cc:824] T 00000000000000000000000000000000 P d0ad74794d8c4019b4179ae77bc11a14: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:51.355154 21431 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0ad74794d8c4019b4179ae77bc11a14: No bootstrap required, opened a new log
I20260928 08:30:51.356025 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:51.342567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41934 (local address 127.15.134.254:33629)
0928 08:30:51.343802 (+ 1235us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:51.343813 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:51.343862 (+ 49us) server_negotiation.cc:407] Connection header received
0928 08:30:51.345866 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:51.345928 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:51.346459 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:51.346818 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:51.347512 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:51.348909 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:51.351008 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:51.351908 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:51.352145 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:51.353486 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:51.353497 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:51.353518 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:30:51.354116 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:51.354178 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:30:51.354205 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:51.354299 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:51.354583 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 08:30:51.354927 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":468,"threads_started":1}
I20260928 08:30:51.360198 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:30:51.366576 21431 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0ad74794d8c4019b4179ae77bc11a14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ad74794d8c4019b4179ae77bc11a14" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 33629 } }
I20260928 08:30:51.366974 21431 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0ad74794d8c4019b4179ae77bc11a14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:51.367069 21431 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0ad74794d8c4019b4179ae77bc11a14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0ad74794d8c4019b4179ae77bc11a14, State: Initialized, Role: FOLLOWER
I20260928 08:30:51.367640 21431 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0ad74794d8c4019b4179ae77bc11a14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ad74794d8c4019b4179ae77bc11a14" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 33629 } }
I20260928 08:30:51.368021 21431 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0ad74794d8c4019b4179ae77bc11a14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:30:51.368237 21431 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0ad74794d8c4019b4179ae77bc11a14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:30:51.368491 21431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0ad74794d8c4019b4179ae77bc11a14 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:51.371140 21431 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0ad74794d8c4019b4179ae77bc11a14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ad74794d8c4019b4179ae77bc11a14" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 33629 } }
I20260928 08:30:51.371781 21431 leader_election.cc:302] T 00000000000000000000000000000000 P d0ad74794d8c4019b4179ae77bc11a14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0ad74794d8c4019b4179ae77bc11a14; no voters:
I20260928 08:30:51.372551 21431 leader_election.cc:288] T 00000000000000000000000000000000 P d0ad74794d8c4019b4179ae77bc11a14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:30:51.372768 21436 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0ad74794d8c4019b4179ae77bc11a14 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:51.373585 21436 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0ad74794d8c4019b4179ae77bc11a14 [term 1 LEADER]: Becoming Leader. State: Replica: d0ad74794d8c4019b4179ae77bc11a14, State: Running, Role: LEADER
I20260928 08:30:51.374127 21436 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0ad74794d8c4019b4179ae77bc11a14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ad74794d8c4019b4179ae77bc11a14" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 33629 } }
I20260928 08:30:51.374928 21431 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0ad74794d8c4019b4179ae77bc11a14 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:30:51.377156 21437 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0ad74794d8c4019b4179ae77bc11a14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0ad74794d8c4019b4179ae77bc11a14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ad74794d8c4019b4179ae77bc11a14" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 33629 } } }
I20260928 08:30:51.377261 21438 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0ad74794d8c4019b4179ae77bc11a14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0ad74794d8c4019b4179ae77bc11a14. Latest consensus state: current_term: 1 leader_uuid: "d0ad74794d8c4019b4179ae77bc11a14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ad74794d8c4019b4179ae77bc11a14" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 33629 } } }
I20260928 08:30:51.377547 21438 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0ad74794d8c4019b4179ae77bc11a14 [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:51.377547 21437 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0ad74794d8c4019b4179ae77bc11a14 [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:51.380182 21444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:30:51.385771 21444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:30:51.394868 21444 catalog_manager.cc:1347] Generated new cluster ID: 7c36a3d390d74893b423443a714a9f08
I20260928 08:30:51.395035 21444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:30:51.409315 21444 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:30:51.410669 21444 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:30:51.417856 21444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0ad74794d8c4019b4179ae77bc11a14: Generated new TSK 0
I20260928 08:30:51.418720 21444 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:30:51.921550 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:51.430530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41942 (local address 127.15.134.254:33629)
0928 08:30:51.430724 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:51.430728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:51.431171 (+ 443us) server_negotiation.cc:407] Connection header received
0928 08:30:51.431422 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:51.431426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:51.431509 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:51.431661 (+ 152us) server_negotiation.cc:226] Negotiated authn=JWT
0928 08:30:51.432237 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:51.433136 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:51.434564 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:51.435015 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:51.435154 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:51.435606 (+ 452us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 08:30:51.920791 (+485185us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 08:30:51.921240 (+ 449us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41942: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":88}
W20260928 08:30:51.921789 21434 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41942: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 08:30:51.921896 21460 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:30:51.430380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:33629 (local address 127.0.0.1:41942)
0928 08:30:51.431067 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:51.431082 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:51.431249 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:51.431676 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:51.431681 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:51.431693 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0928 08:30:51.431965 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:51.431973 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:51.433292 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:51.433304 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:51.434421 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:51.434432 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:51.435187 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:51.435193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:51.435333 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:51.435365 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 08:30:51.921157 (+485792us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 08:30:51.921465 (+ 308us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.15.134.254:33629: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":407,"threads_started":1}
I20260928 08:30:51.923054 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 21365
I20260928 08:30:51.924136 21426 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:52.028210 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 21365
2026-09-28T08:30:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1198 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (437 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (441 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:30:53 dist-test-slave-qp2n krb5kdc[21475](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:30:53 dist-test-slave-qp2n krb5kdc[21475](info): set up 2 sockets
Sep 28 08:30:53 dist-test-slave-qp2n krb5kdc[21475](info): commencing operation
krb5kdc: starting...
W20260928 08:30:54.007026 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:30:54 dist-test-slave-qp2n krb5kdc[21475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:30:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:30:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:30:54.225170 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:35077
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:46291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:35077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:54.567829 21488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:54.568210 21488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:54.568477 21488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:54.576140 21488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:30:54.576313 21488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:54.576440 21488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:54.576540 21488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:30:54.576646 21488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:30:54.586972 21488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:46291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:35077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:35077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:54.588408 21488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:54.589813 21488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:54.601970 21493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:54.602047 21496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:54.602073 21494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:54.603147 21488 server_base.cc:1034] running on GCE node
I20260928 08:30:54.604107 21488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:54.605867 21488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:54.607089 21488 hybrid_clock.cc:648] HybridClock initialized: now 1790584254607048 us; error 40 us; skew 500 ppm
I20260928 08:30:54.607481 21488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:54 dist-test-slave-qp2n krb5kdc[21475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584254, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:54.610795 21488 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:30:54.612397 21488 webserver.cc:460] Webserver started at http://127.15.134.254:33521/ using document root <none> and password file <none>
I20260928 08:30:54.613173 21488 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:54.613307 21488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:54.613651 21488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:54.616881 21488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "c8aa8c47240c4d16addadd829b6cfd4b"
format_stamp: "Formatted at 2026-09-28 08:30:54 on dist-test-slave-qp2n"
server_key: "a65bf87b1bb3c951629d292993989b31"
server_key_iv: "67177cf306045246911f3d5d9074dd5b"
server_key_version: "encryptionkey@0"
I20260928 08:30:54.617708 21488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "c8aa8c47240c4d16addadd829b6cfd4b"
format_stamp: "Formatted at 2026-09-28 08:30:54 on dist-test-slave-qp2n"
server_key: "a65bf87b1bb3c951629d292993989b31"
server_key_iv: "67177cf306045246911f3d5d9074dd5b"
server_key_version: "encryptionkey@0"
I20260928 08:30:54.623131 21488 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 08:30:54.626726 21503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:54.628194 21488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:30:54.628330 21488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "c8aa8c47240c4d16addadd829b6cfd4b"
format_stamp: "Formatted at 2026-09-28 08:30:54 on dist-test-slave-qp2n"
server_key: "a65bf87b1bb3c951629d292993989b31"
server_key_iv: "67177cf306045246911f3d5d9074dd5b"
server_key_version: "encryptionkey@0"
I20260928 08:30:54.628499 21488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:54.649439 21488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:54.653687 21488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:54.653934 21488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:54.673018 21488 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:35077
I20260928 08:30:54.673064 21554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:35077 every 8 connection(s)
I20260928 08:30:54.674531 21488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:30:54.676517 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 21488
I20260928 08:30:54.676751 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:30:54.677409 15899 external_mini_cluster.cc:1428] Setting key 8c71d2513199e37b48b70303b9b2b11b
I20260928 08:30:54.678701 21555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:54.690728 21555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8aa8c47240c4d16addadd829b6cfd4b: Bootstrap starting.
Sep 28 08:30:54 dist-test-slave-qp2n krb5kdc[21475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:54.694239 21555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8aa8c47240c4d16addadd829b6cfd4b: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:54.695393 21555 log.cc:824] T 00000000000000000000000000000000 P c8aa8c47240c4d16addadd829b6cfd4b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:54.698541 21555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8aa8c47240c4d16addadd829b6cfd4b: No bootstrap required, opened a new log
I20260928 08:30:54.700090 21558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:54.679434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45954 (local address 127.15.134.254:35077)
0928 08:30:54.680168 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:54.680181 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:54.680220 (+ 39us) server_negotiation.cc:407] Connection header received
0928 08:30:54.681938 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:54.681960 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:54.682365 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:54.682809 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:54.684315 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:54.685732 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:54.687038 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:54.687797 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:54.688020 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:54.692023 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:54.692032 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:54.692048 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:54.692093 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:54.695369 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:54.696119 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:54.696135 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:54.696142 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:54.696321 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:54.697044 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:54.697064 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:54.697067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:54.697641 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:54.698014 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:54.698467 (+ 453us) server_negotiation.cc:299] Negotiation successful
0928 08:30:54.698980 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":318,"threads_started":1}
I20260928 08:30:54.706799 21555 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8aa8c47240c4d16addadd829b6cfd4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8aa8c47240c4d16addadd829b6cfd4b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35077 } }
I20260928 08:30:54.707199 21555 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8aa8c47240c4d16addadd829b6cfd4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:54.707315 21555 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8aa8c47240c4d16addadd829b6cfd4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8aa8c47240c4d16addadd829b6cfd4b, State: Initialized, Role: FOLLOWER
I20260928 08:30:54.707903 21555 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8aa8c47240c4d16addadd829b6cfd4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8aa8c47240c4d16addadd829b6cfd4b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35077 } }
I20260928 08:30:54.708158 21555 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8aa8c47240c4d16addadd829b6cfd4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:30:54.708309 21555 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8aa8c47240c4d16addadd829b6cfd4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:30:54.708498 21555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8aa8c47240c4d16addadd829b6cfd4b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:54.710713 21555 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8aa8c47240c4d16addadd829b6cfd4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8aa8c47240c4d16addadd829b6cfd4b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35077 } }
I20260928 08:30:54.711277 21555 leader_election.cc:302] T 00000000000000000000000000000000 P c8aa8c47240c4d16addadd829b6cfd4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8aa8c47240c4d16addadd829b6cfd4b; no voters:
I20260928 08:30:54.712071 21555 leader_election.cc:288] T 00000000000000000000000000000000 P c8aa8c47240c4d16addadd829b6cfd4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:30:54.712216 21560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8aa8c47240c4d16addadd829b6cfd4b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:54.714205 21555 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8aa8c47240c4d16addadd829b6cfd4b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:30:54.714679 21560 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8aa8c47240c4d16addadd829b6cfd4b [term 1 LEADER]: Becoming Leader. State: Replica: c8aa8c47240c4d16addadd829b6cfd4b, State: Running, Role: LEADER
I20260928 08:30:54.715222 21560 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8aa8c47240c4d16addadd829b6cfd4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8aa8c47240c4d16addadd829b6cfd4b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35077 } }
I20260928 08:30:54.718390 21562 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8aa8c47240c4d16addadd829b6cfd4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8aa8c47240c4d16addadd829b6cfd4b. Latest consensus state: current_term: 1 leader_uuid: "c8aa8c47240c4d16addadd829b6cfd4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8aa8c47240c4d16addadd829b6cfd4b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35077 } } }
I20260928 08:30:54.718802 21562 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8aa8c47240c4d16addadd829b6cfd4b [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:54.719307 21561 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8aa8c47240c4d16addadd829b6cfd4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8aa8c47240c4d16addadd829b6cfd4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8aa8c47240c4d16addadd829b6cfd4b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35077 } } }
I20260928 08:30:54.719617 21561 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8aa8c47240c4d16addadd829b6cfd4b [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:54.719938 21572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:30:54.726763 21572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:30:54.735941 21572 catalog_manager.cc:1347] Generated new cluster ID: 30c6ae1a46134cbda30bf58fb5db48c1
I20260928 08:30:54.736073 21572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:30:54.744554 21572 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:30:54.745999 21572 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:30:54.770339 21572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8aa8c47240c4d16addadd829b6cfd4b: Generated new TSK 0
I20260928 08:30:54.771322 21572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.134.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
I20260928 08:30:54.829442 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:0
--local_ip_for_outbound_sockets=127.15.134.193
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:35077
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:46291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:55.172266 21581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:55.172561 21581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:55.172745 21581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:55.180140 21581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:55.180281 21581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:55.180485 21581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:30:55.190250 21581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:46291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:35077
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:55.191591 21581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:55.192951 21581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:55.206727 21586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:55.206790 21589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:55.206815 21587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:55.207583 21581 server_base.cc:1034] running on GCE node
I20260928 08:30:55.208216 21581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:55.209275 21581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:55.210525 21581 hybrid_clock.cc:648] HybridClock initialized: now 1790584255210470 us; error 46 us; skew 500 ppm
I20260928 08:30:55.210970 21581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:55 dist-test-slave-qp2n krb5kdc[21475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584255, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:55.217512 21581 init.cc:377] Logged in from keytab as kudu/127.15.134.193@KRBTEST.COM (short username kudu)
I20260928 08:30:55.219216 21581 webserver.cc:460] Webserver started at http://127.15.134.193:36709/ using document root <none> and password file <none>
I20260928 08:30:55.220013 21581 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:55.220145 21581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:55.220418 21581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:55.223567 21581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-0/data/instance:
uuid: "472a52ae97d84bdca676a1e7cb04bf10"
format_stamp: "Formatted at 2026-09-28 08:30:55 on dist-test-slave-qp2n"
server_key: "960e7c03a49dd1d4b7b9e6a9eff7aab7"
server_key_iv: "57077812361c83079024ab0d458c8169"
server_key_version: "encryptionkey@0"
I20260928 08:30:55.224336 21581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance:
uuid: "472a52ae97d84bdca676a1e7cb04bf10"
format_stamp: "Formatted at 2026-09-28 08:30:55 on dist-test-slave-qp2n"
server_key: "960e7c03a49dd1d4b7b9e6a9eff7aab7"
server_key_iv: "57077812361c83079024ab0d458c8169"
server_key_version: "encryptionkey@0"
I20260928 08:30:55.230512 21581 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 08:30:55.234615 21596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:55.236186 21581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:30:55.236373 21581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "472a52ae97d84bdca676a1e7cb04bf10"
format_stamp: "Formatted at 2026-09-28 08:30:55 on dist-test-slave-qp2n"
server_key: "960e7c03a49dd1d4b7b9e6a9eff7aab7"
server_key_iv: "57077812361c83079024ab0d458c8169"
server_key_version: "encryptionkey@0"
I20260928 08:30:55.236665 21581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:55.256269 21581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:55.261482 21581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:55.261828 21581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:55.262921 21581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:30:55.264403 21581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:55.264477 21581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:55.264675 21581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:55.264744 21581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260928 08:30:55.301234 21581 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:43945
I20260928 08:30:55.301270 21708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:43945 every 8 connection(s)
I20260928 08:30:55.302575 21581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:30:55.302951 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 21581
I20260928 08:30:55.303149 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance
I20260928 08:30:55.304020 15899 external_mini_cluster.cc:1428] Setting key bc2456298eb7fbfe9d93cc83c5dd809d
Sep 28 08:30:55 dist-test-slave-qp2n krb5kdc[21475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584255, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:55.327661 21711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:55.305394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:39725 (local address 127.15.134.254:35077)
0928 08:30:55.305893 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:55.305900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:55.307101 (+ 1201us) server_negotiation.cc:407] Connection header received
0928 08:30:55.310027 (+ 2926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:55.310031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:55.310103 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:55.310211 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:55.312316 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:55.313165 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:55.314954 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:55.315886 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:55.316052 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:55.320129 (+ 4077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:55.320139 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:55.320147 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:55.320199 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:55.324074 (+ 3875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:55.324902 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:55.324910 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:55.324917 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:55.325090 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:55.325534 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:55.325542 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:55.325546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:55.325986 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:55.326156 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:55.326873 (+ 717us) server_negotiation.cc:299] Negotiation successful
0928 08:30:55.327139 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":239,"threads_started":1}
WARNING: no policy specified for kudu/127.15.134.194@KRBTEST.COM; defaulting to no policy
I20260928 08:30:55.328372 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:55.306058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:35077 (local address 127.15.134.193:39725)
0928 08:30:55.306917 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:55.306969 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:55.309695 (+ 2726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:55.310484 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:55.310498 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:55.311268 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:55.312097 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:55.312125 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:55.313335 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:55.313343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:55.314806 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:55.314825 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:55.316113 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:55.316118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:55.316315 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:55.317220 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:55.317244 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:55.319868 (+ 2624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:55.324233 (+ 4365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:55.324241 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:55.324255 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:55.324773 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:55.325213 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:55.325216 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:55.325218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:55.325390 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:55.326114 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:55.326120 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:55.326447 (+ 327us) client_negotiation.cc:769] Sending connection context
0928 08:30:55.326672 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 08:30:55.327085 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":315,"threads_started":1}
I20260928 08:30:55.330188 21709 heartbeater.cc:347] Connected to a master server at 127.15.134.254:35077
I20260928 08:30:55.330549 21709 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.194@KRBTEST.COM" created.
I20260928 08:30:55.331609 21709 heartbeater.cc:511] Master 127.15.134.254:35077 requested a full tablet report, sending...
I20260928 08:30:55.334188 21520 ts_manager.cc:194] Registered new tserver with Master: 472a52ae97d84bdca676a1e7cb04bf10 (127.15.134.193:43945)
I20260928 08:30:55.335776 21520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.193@KRBTEST.COM'} at 127.15.134.193:39725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
I20260928 08:30:55.360473 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:0
--local_ip_for_outbound_sockets=127.15.134.194
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:35077
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:46291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:55.720810 21715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:55.721146 21715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:55.721323 21715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:55.728523 21715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:55.728694 21715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:55.728878 21715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:30:55.739266 21715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:46291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:35077
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:55.740772 21715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:55.742224 21715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:55.756338 21721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:55.756413 21723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:55.756970 21720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:55.757302 21715 server_base.cc:1034] running on GCE node
I20260928 08:30:55.757916 21715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:55.758898 21715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:55.760215 21715 hybrid_clock.cc:648] HybridClock initialized: now 1790584255760163 us; error 42 us; skew 500 ppm
I20260928 08:30:55.760740 21715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:55 dist-test-slave-qp2n krb5kdc[21475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584255, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:55.764287 21715 init.cc:377] Logged in from keytab as kudu/127.15.134.194@KRBTEST.COM (short username kudu)
I20260928 08:30:55.766057 21715 webserver.cc:460] Webserver started at http://127.15.134.194:38779/ using document root <none> and password file <none>
I20260928 08:30:55.766803 21715 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:55.766919 21715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:55.767196 21715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:55.770710 21715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-1/data/instance:
uuid: "a70590f00c114c7781ccf61860198c61"
format_stamp: "Formatted at 2026-09-28 08:30:55 on dist-test-slave-qp2n"
server_key: "9784cadedd53370893f05a5b5bc175bc"
server_key_iv: "22bd5c07a597ab1c8d693c0722609829"
server_key_version: "encryptionkey@0"
I20260928 08:30:55.771641 21715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance:
uuid: "a70590f00c114c7781ccf61860198c61"
format_stamp: "Formatted at 2026-09-28 08:30:55 on dist-test-slave-qp2n"
server_key: "9784cadedd53370893f05a5b5bc175bc"
server_key_iv: "22bd5c07a597ab1c8d693c0722609829"
server_key_version: "encryptionkey@0"
I20260928 08:30:55.777956 21715 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 08:30:55.781945 21730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:55.783644 21715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:30:55.783864 21715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "a70590f00c114c7781ccf61860198c61"
format_stamp: "Formatted at 2026-09-28 08:30:55 on dist-test-slave-qp2n"
server_key: "9784cadedd53370893f05a5b5bc175bc"
server_key_iv: "22bd5c07a597ab1c8d693c0722609829"
server_key_version: "encryptionkey@0"
I20260928 08:30:55.784108 21715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:55.804778 21715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:55.809443 21715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:55.809808 21715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:55.810859 21715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:30:55.812355 21715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:55.812469 21715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:55.812674 21715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:55.812763 21715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260928 08:30:55.854619 21715 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:33843
I20260928 08:30:55.854662 21842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:33843 every 8 connection(s)
I20260928 08:30:55.856046 21715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:30:55.863346 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 21715
I20260928 08:30:55.863525 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance
I20260928 08:30:55.864089 15899 external_mini_cluster.cc:1428] Setting key bdaee0f4f7791d22b9da707171eb5f96
Sep 28 08:30:55 dist-test-slave-qp2n krb5kdc[21475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584255, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:55.879500 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:55.858942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:55795 (local address 127.15.134.254:35077)
0928 08:30:55.859409 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:55.859415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:55.860304 (+ 889us) server_negotiation.cc:407] Connection header received
0928 08:30:55.862201 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:55.862206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:55.862302 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:55.862429 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:55.864512 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:55.865561 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:55.867902 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:55.868395 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:55.868502 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:55.872994 (+ 4492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:55.873000 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:55.873004 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:55.873041 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:55.875926 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:55.876830 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:55.876851 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:55.876854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:55.876996 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:55.877512 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:55.877517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:55.877520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:55.877816 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:55.878011 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:55.878811 (+ 800us) server_negotiation.cc:299] Negotiation successful
0928 08:30:55.879075 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":255,"threads_started":1}
I20260928 08:30:55.880671 21846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:55.859393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:35077 (local address 127.15.134.194:55795)
0928 08:30:55.860135 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:55.860173 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:55.861918 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:55.862692 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:55.862700 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:55.863465 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:55.864295 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:55.864322 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:55.865728 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:55.865732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:55.867725 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:55.867744 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:55.868665 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:55.868676 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:55.869008 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:55.869815 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:55.869854 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:55.872719 (+ 2865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:55.876097 (+ 3378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:55.876107 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:55.876128 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:55.876681 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:55.877135 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:55.877141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:55.877145 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:55.877370 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:55.877971 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:55.877980 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:55.878409 (+ 429us) client_negotiation.cc:769] Sending connection context
0928 08:30:55.878712 (+ 303us) client_negotiation.cc:240] Negotiation successful
0928 08:30:55.879319 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":308,"threads_started":1}
I20260928 08:30:55.882498 21843 heartbeater.cc:347] Connected to a master server at 127.15.134.254:35077
I20260928 08:30:55.882843 21843 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:55.883816 21843 heartbeater.cc:511] Master 127.15.134.254:35077 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.134.195@KRBTEST.COM; defaulting to no policy
I20260928 08:30:55.885629 21520 ts_manager.cc:194] Registered new tserver with Master: a70590f00c114c7781ccf61860198c61 (127.15.134.194:33843)
I20260928 08:30:55.886497 21520 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.194@KRBTEST.COM'} at 127.15.134.194:55795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
I20260928 08:30:55.913208 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:0
--local_ip_for_outbound_sockets=127.15.134.195
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:35077
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:46291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:56.246383 21849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:56.246706 21849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:56.246870 21849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:56.254185 21849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:56.254359 21849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:56.254562 21849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:30:56.264743 21849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:46291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:35077
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:56.266132 21849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:56.267498 21849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:56.281122 21855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:56.281078 21854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:56.281054 21857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:56.281980 21849 server_base.cc:1034] running on GCE node
I20260928 08:30:56.282569 21849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:56.283614 21849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:56.284914 21849 hybrid_clock.cc:648] HybridClock initialized: now 1790584256284836 us; error 62 us; skew 500 ppm
I20260928 08:30:56.285287 21849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:56 dist-test-slave-qp2n krb5kdc[21475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584256, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:56.288563 21849 init.cc:377] Logged in from keytab as kudu/127.15.134.195@KRBTEST.COM (short username kudu)
I20260928 08:30:56.290271 21849 webserver.cc:460] Webserver started at http://127.15.134.195:44227/ using document root <none> and password file <none>
I20260928 08:30:56.291029 21849 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:56.291184 21849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:56.291503 21849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:56.294670 21849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-2/data/instance:
uuid: "ebb18b942d5b46269ac21090093503fe"
format_stamp: "Formatted at 2026-09-28 08:30:56 on dist-test-slave-qp2n"
server_key: "26f10fbae8bcbe73028c428523279fb6"
server_key_iv: "6c39f01722bf485715c706f25ff0b967"
server_key_version: "encryptionkey@0"
I20260928 08:30:56.295485 21849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance:
uuid: "ebb18b942d5b46269ac21090093503fe"
format_stamp: "Formatted at 2026-09-28 08:30:56 on dist-test-slave-qp2n"
server_key: "26f10fbae8bcbe73028c428523279fb6"
server_key_iv: "6c39f01722bf485715c706f25ff0b967"
server_key_version: "encryptionkey@0"
I20260928 08:30:56.300772 21849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 08:30:56.304440 21864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:56.305933 21849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:30:56.306113 21849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "ebb18b942d5b46269ac21090093503fe"
format_stamp: "Formatted at 2026-09-28 08:30:56 on dist-test-slave-qp2n"
server_key: "26f10fbae8bcbe73028c428523279fb6"
server_key_iv: "6c39f01722bf485715c706f25ff0b967"
server_key_version: "encryptionkey@0"
I20260928 08:30:56.306314 21849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:56.338124 21849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:56.339563 21709 heartbeater.cc:503] Master 127.15.134.254:35077 was elected leader, sending a full tablet report...
I20260928 08:30:56.342651 21849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:56.342877 21849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:56.343755 21849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:30:56.345155 21849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:56.345238 21849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:56.345323 21849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:56.345366 21849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:56.377892 21849 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:40737
I20260928 08:30:56.377941 21976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:40737 every 8 connection(s)
I20260928 08:30:56.379199 21849 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:30:56.384845 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 21849
I20260928 08:30:56.385010 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance
I20260928 08:30:56.385598 15899 external_mini_cluster.cc:1428] Setting key 0cdb2590c296945928a668af090db59c
Sep 28 08:30:56 dist-test-slave-qp2n krb5kdc[21475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584256, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:30:56.399924 21979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:56.381910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:36825 (local address 127.15.134.254:35077)
0928 08:30:56.382335 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:56.382341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:56.383357 (+ 1016us) server_negotiation.cc:407] Connection header received
0928 08:30:56.385300 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:56.385306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:56.385394 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:56.385521 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:56.387114 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:56.388072 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:56.389941 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:56.390356 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:56.390492 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:56.393895 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:56.393901 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:56.393904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:56.393947 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:56.396454 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:56.397261 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:56.397265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:56.397267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:56.397429 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:56.397911 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:56.397915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:56.397918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:56.398223 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:56.398363 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:56.399048 (+ 685us) server_negotiation.cc:299] Negotiation successful
0928 08:30:56.399424 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":191,"threads_started":1}
I20260928 08:30:56.400799 21980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:56.382494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:35077 (local address 127.15.134.195:36825)
0928 08:30:56.383194 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:56.383238 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:56.384971 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:56.385767 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:56.385776 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:56.386288 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:56.386974 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:56.386994 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:56.388216 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:56.388222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:56.389774 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:56.389787 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:56.390580 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:56.390584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:56.390781 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:56.391434 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:56.391461 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:56.393682 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:56.396592 (+ 2910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:56.396599 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:56.396659 (+ 60us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:56.397143 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:56.397528 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:56.397532 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:56.397534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:56.397712 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:56.398347 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:56.398353 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:56.398691 (+ 338us) client_negotiation.cc:769] Sending connection context
0928 08:30:56.398928 (+ 237us) client_negotiation.cc:240] Negotiation successful
0928 08:30:56.399337 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":270,"threads_started":1}
I20260928 08:30:56.402251 21977 heartbeater.cc:347] Connected to a master server at 127.15.134.254:35077
I20260928 08:30:56.402588 21977 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:56.403365 21977 heartbeater.cc:511] Master 127.15.134.254:35077 requested a full tablet report, sending...
I20260928 08:30:56.405056 21519 ts_manager.cc:194] Registered new tserver with Master: ebb18b942d5b46269ac21090093503fe (127.15.134.195:40737)
I20260928 08:30:56.406061 21519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.195@KRBTEST.COM'} at 127.15.134.195:36825
I20260928 08:30:56.414582 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/test-admin.keytab.
Sep 28 08:30:56 dist-test-slave-qp2n krb5kdc[21475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:56.446033 15899 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 08:30:56.451555 21985 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:30:56.448892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42292 (local address 127.15.134.193:43945)
0928 08:30:56.449580 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:56.449587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:56.449614 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:30:56.449676 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:56.449683 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:56.449901 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:56.450032 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:56.450989 (+ 957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42292: BlockingRecv error: recv got EOF from 127.0.0.1:42292 (error 108)
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":330,"threads_started":1}
W20260928 08:30:56.451798 15899 init.cc:244] Error while opening credential cache '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/krb5cc)
Sep 28 08:30:56 dist-test-slave-qp2n krb5kdc[21475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:56.453554 15899 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 08:30:56 dist-test-slave-qp2n krb5kdc[21475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:56.454973 15899 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 08:30:56 dist-test-slave-qp2n krb5kdc[21475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.193@KRBTEST.COM
I20260928 08:30:56.470381 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:56.456547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42298 (local address 127.15.134.193:43945)
0928 08:30:56.456853 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:56.456860 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:56.457116 (+ 256us) server_negotiation.cc:407] Connection header received
0928 08:30:56.457240 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:56.457244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:56.457307 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:56.457419 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:56.458399 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:56.459396 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:56.460597 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:56.461280 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:56.461385 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:56.464946 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:56.464959 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:56.464975 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:56.465011 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:56.467731 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:56.468386 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:56.468396 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:56.468398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:56.468516 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:56.468926 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:56.468930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:56.468934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:56.469291 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:56.469537 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:56.469791 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 08:30:56.470081 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
W20260928 08:30:56.476514 21985 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:30:56.474677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42312 (local address 127.15.134.193:43945)
0928 08:30:56.474883 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:56.474890 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:56.475105 (+ 215us) server_negotiation.cc:407] Connection header received
0928 08:30:56.475299 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:56.475304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:56.475372 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:56.475477 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:56.476253 (+ 776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42312: BlockingRecv error: recv got EOF from 127.0.0.1:42312 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 08:30:56.476728 15899 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/krb5cc' of type FILE
Sep 28 08:30:56 dist-test-slave-qp2n krb5kdc[21475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:56.478284 15899 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 08:30:56 dist-test-slave-qp2n krb5kdc[21475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:56.479581 15899 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 08:30:56 dist-test-slave-qp2n krb5kdc[21475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.193@KRBTEST.COM
I20260928 08:30:56.495064 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:56.481012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42314 (local address 127.15.134.193:43945)
0928 08:30:56.481241 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:56.481250 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:56.481584 (+ 334us) server_negotiation.cc:407] Connection header received
0928 08:30:56.481776 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:56.481782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:56.481855 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:56.481962 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:56.482947 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:56.483657 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:56.485036 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:56.485497 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:56.485641 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:56.489606 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:56.489612 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:56.489615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:56.489665 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:56.492309 (+ 2644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:56.493137 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:56.493143 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:56.493152 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:56.493281 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:56.493697 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:56.493702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:56.493704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:56.493995 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:56.494270 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:56.494564 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 08:30:56.494770 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260928 08:30:56.499505 21985 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:30:56.497752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42330 (local address 127.15.134.193:43945)
0928 08:30:56.497954 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:56.497961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:56.498242 (+ 281us) server_negotiation.cc:407] Connection header received
0928 08:30:56.498422 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:56.498427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:56.498505 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:56.498601 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:56.499272 (+ 671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42330: BlockingRecv error: recv got EOF from 127.0.0.1:42330 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 08:30:56.499799 15899 init.cc:244] Error while opening credential cache '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790584205252592-15899-0/krb5kdc/krb5cc)
Sep 28 08:30:56 dist-test-slave-qp2n krb5kdc[21475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:56.501313 15899 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 08:30:56 dist-test-slave-qp2n krb5kdc[21475](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:56.502573 15899 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 08:30:56 dist-test-slave-qp2n krb5kdc[21475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.193@KRBTEST.COM
I20260928 08:30:56.517017 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:56.503882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42338 (local address 127.15.134.193:43945)
0928 08:30:56.504085 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:56.504092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:56.504276 (+ 184us) server_negotiation.cc:407] Connection header received
0928 08:30:56.504451 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:56.504456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:56.504522 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:56.504617 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:56.505561 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:56.506273 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:56.507480 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:56.507920 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:56.508047 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:56.511592 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:56.511598 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:56.511603 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:56.511641 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:56.514152 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:56.514989 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:56.514993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:56.514995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:56.515139 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:56.515593 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:56.515596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:56.515598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:56.515897 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:56.516124 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:56.516398 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 08:30:56.516570 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:30:56.519016 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 21581
I20260928 08:30:56.531417 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:56.519657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42346 (local address 127.15.134.193:43945)
0928 08:30:56.519843 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:56.519851 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:56.520192 (+ 341us) server_negotiation.cc:407] Connection header received
0928 08:30:56.520306 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:56.520309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:56.520361 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:56.520473 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:56.521487 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:56.522133 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:56.523502 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:56.523916 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:56.524007 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:56.526349 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:56.526353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:56.526357 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:56.526392 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:56.528858 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:56.529553 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:56.529557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:56.529560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:56.529690 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:56.530027 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:56.530030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:56.530032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:56.530269 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:56.530487 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:56.530774 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 08:30:56.530936 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 08:30:56.532068 21704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:56.620235 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 21581
I20260928 08:30:56.637555 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 21715
Sep 28 08:30:56 dist-test-slave-qp2n krb5kdc[21475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.194@KRBTEST.COM
I20260928 08:30:56.654491 21998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:56.638540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59606 (local address 127.15.134.194:33843)
0928 08:30:56.639177 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:56.639185 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:56.639211 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:30:56.639272 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:56.639279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:56.639492 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:56.639626 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:56.640709 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:56.641709 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:56.642936 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:56.643488 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:56.643589 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:56.647837 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:56.647852 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:56.647872 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:56.647917 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:56.651135 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:56.651943 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:56.651951 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:56.651954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:56.652106 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:56.652735 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:56.652739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:56.652742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:56.653151 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:56.653515 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:56.653741 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 08:30:56.654029 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":368,"threads_started":1}
I20260928 08:30:56.655921 21838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:56.742597 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 21715
I20260928 08:30:56.759939 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 21849
Sep 28 08:30:56 dist-test-slave-qp2n krb5kdc[21475](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.195@KRBTEST.COM
I20260928 08:30:56.776942 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:56.761211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59534 (local address 127.15.134.195:40737)
0928 08:30:56.761908 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:56.761915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:56.761949 (+ 34us) server_negotiation.cc:407] Connection header received
0928 08:30:56.762012 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:56.762018 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:56.762249 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:56.762417 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:56.763563 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:56.764702 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:56.766036 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:56.766661 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:56.766779 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:56.771007 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:56.771021 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:56.771038 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:56.771086 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:56.773928 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:56.774654 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:56.774661 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:56.774663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:56.774785 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:56.775229 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:56.775232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:56.775235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:56.775658 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:56.775922 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:56.776249 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 08:30:56.776541 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":343,"threads_started":1}
I20260928 08:30:56.778301 21972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:56.862565 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 21849
I20260928 08:30:56.879029 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 21488
I20260928 08:30:56.880278 21550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:30:56.984820 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 21488
2026-09-28T08:30:57Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4082 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:30:57 dist-test-slave-qp2n krb5kdc[22005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:30:57 dist-test-slave-qp2n krb5kdc[22005](info): set up 2 sockets
Sep 28 08:30:57 dist-test-slave-qp2n krb5kdc[22005](info): commencing operation
krb5kdc: starting...
W20260928 08:30:58.091045 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.002s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:30:58 dist-test-slave-qp2n krb5kdc[22005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:30:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:30:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.254.keytab.
Entry for principal oryx/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.254.keytab.
I20260928 08:30:58.314492 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:44257
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.254.keytab
--principal=oryx/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:46403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:44257
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:58.657598 22018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:58.657976 22018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:58.658133 22018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:58.665644 22018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:30:58.665807 22018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:58.665912 22018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:58.666024 22018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:30:58.666101 22018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:30:58.676282 22018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:46403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:44257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.254.keytab
--principal=oryx/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:44257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:58.677788 22018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:58.679155 22018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:58.691255 22026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:58.691228 22024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:58.691308 22023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:58.693064 22018 server_base.cc:1034] running on GCE node
I20260928 08:30:58.693888 22018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:58.695538 22018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:58.696769 22018 hybrid_clock.cc:648] HybridClock initialized: now 1790584258696718 us; error 37 us; skew 500 ppm
I20260928 08:30:58.697161 22018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:58 dist-test-slave-qp2n krb5kdc[22005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584258, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:58.700270 22018 init.cc:377] Logged in from keytab as oryx/127.15.134.254@KRBTEST.COM (short username oryx)
I20260928 08:30:58.701877 22018 webserver.cc:460] Webserver started at http://127.15.134.254:38541/ using document root <none> and password file <none>
I20260928 08:30:58.702692 22018 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:58.702853 22018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:58.703166 22018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:58.706365 22018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "7d3e3d1d841b4840ae24988248358275"
format_stamp: "Formatted at 2026-09-28 08:30:58 on dist-test-slave-qp2n"
server_key: "8b01595b0bdaa986caa4dc5f8ef7dc47"
server_key_iv: "9c927506c8bcb2ea5297d38b516daf36"
server_key_version: "encryptionkey@0"
I20260928 08:30:58.707116 22018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "7d3e3d1d841b4840ae24988248358275"
format_stamp: "Formatted at 2026-09-28 08:30:58 on dist-test-slave-qp2n"
server_key: "8b01595b0bdaa986caa4dc5f8ef7dc47"
server_key_iv: "9c927506c8bcb2ea5297d38b516daf36"
server_key_version: "encryptionkey@0"
I20260928 08:30:58.712193 22018 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:30:58.715605 22033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:58.717089 22018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:30:58.717280 22018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "7d3e3d1d841b4840ae24988248358275"
format_stamp: "Formatted at 2026-09-28 08:30:58 on dist-test-slave-qp2n"
server_key: "8b01595b0bdaa986caa4dc5f8ef7dc47"
server_key_iv: "9c927506c8bcb2ea5297d38b516daf36"
server_key_version: "encryptionkey@0"
I20260928 08:30:58.717499 22018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:58.739022 22018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:58.743206 22018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:58.743507 22018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:58.762549 22018 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:44257
I20260928 08:30:58.762569 22084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:44257 every 8 connection(s)
I20260928 08:30:58.763998 22018 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:30:58.766639 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 22018
I20260928 08:30:58.766873 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:30:58.767695 15899 external_mini_cluster.cc:1428] Setting key a12b737121f083ace08ef675a4ddf66d
I20260928 08:30:58.768558 22085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:30:58.783740 22085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275: Bootstrap starting.
Sep 28 08:30:58 dist-test-slave-qp2n krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM
I20260928 08:30:58.787916 22085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275: Neither blocks nor log segments found. Creating new log.
I20260928 08:30:58.789376 22085 log.cc:824] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275: Log is configured to *not* fsync() on all Append() calls
I20260928 08:30:58.793323 22085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275: No bootstrap required, opened a new log
I20260928 08:30:58.794895 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:58.770054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51508 (local address 127.15.134.254:44257)
0928 08:30:58.771090 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:58.771103 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:58.771154 (+ 51us) server_negotiation.cc:407] Connection header received
0928 08:30:58.773598 (+ 2444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:58.773623 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:58.774112 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:58.774515 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:58.776410 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:58.777887 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:58.779589 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:58.780315 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:58.780488 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:58.785391 (+ 4903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:58.785410 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:58.785433 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:58.785503 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:58.789682 (+ 4179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:58.790697 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:58.790705 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:58.790712 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:58.790933 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:58.791427 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:58.791432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:58.791435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:58.792154 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:58.792505 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:58.792957 (+ 452us) server_negotiation.cc:299] Negotiation successful
0928 08:30:58.793431 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":548,"threads_started":1}
I20260928 08:30:58.802393 22085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } }
I20260928 08:30:58.802775 22085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:30:58.802860 22085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d3e3d1d841b4840ae24988248358275, State: Initialized, Role: FOLLOWER
I20260928 08:30:58.803422 22085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } }
I20260928 08:30:58.803632 22085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:30:58.803731 22085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:30:58.803869 22085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:30:58.806124 22085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } }
I20260928 08:30:58.806670 22085 leader_election.cc:302] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d3e3d1d841b4840ae24988248358275; no voters:
I20260928 08:30:58.807426 22085 leader_election.cc:288] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:30:58.807561 22090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:30:58.808284 22090 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [term 1 LEADER]: Becoming Leader. State: Replica: 7d3e3d1d841b4840ae24988248358275, State: Running, Role: LEADER
I20260928 08:30:58.808995 22090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } }
I20260928 08:30:58.810016 22085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:30:58.812502 22091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d3e3d1d841b4840ae24988248358275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } }
I20260928 08:30:58.812980 22091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:58.813619 22092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d3e3d1d841b4840ae24988248358275. Latest consensus state: current_term: 1 leader_uuid: "7d3e3d1d841b4840ae24988248358275" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } }
I20260928 08:30:58.815258 22092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [sys.catalog]: This master's current role is: LEADER
I20260928 08:30:58.816560 22097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:30:58.822824 22097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:30:58.831584 22097 catalog_manager.cc:1347] Generated new cluster ID: a09d68fb4c9f429dbd7b25f7f52fd6ec
I20260928 08:30:58.831712 22097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:30:58.845091 22097 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:30:58.846474 22097 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:30:58.854202 22097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275: Generated new TSK 0
I20260928 08:30:58.855087 22097 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.134.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.134.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.134.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.193.keytab.
Entry for principal oryx/127.15.134.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.193.keytab.
I20260928 08:30:58.916904 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:0
--local_ip_for_outbound_sockets=127.15.134.193
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:44257
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.193.keytab
--principal=oryx/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:46403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:59.253490 22111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:59.253840 22111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:59.254011 22111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:59.261530 22111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:59.261698 22111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:59.261888 22111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:30:59.271790 22111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:46403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.193.keytab
--principal=oryx/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:44257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:59.273308 22111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:59.274641 22111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:59.287850 22116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:59.287904 22117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:59.287925 22119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:59.289259 22111 server_base.cc:1034] running on GCE node
I20260928 08:30:59.289945 22111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:59.290990 22111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:59.292227 22111 hybrid_clock.cc:648] HybridClock initialized: now 1790584259292187 us; error 43 us; skew 500 ppm
I20260928 08:30:59.292565 22111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:59 dist-test-slave-qp2n krb5kdc[22005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584259, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:59.296113 22111 init.cc:377] Logged in from keytab as oryx/127.15.134.193@KRBTEST.COM (short username oryx)
I20260928 08:30:59.297888 22111 webserver.cc:460] Webserver started at http://127.15.134.193:35557/ using document root <none> and password file <none>
I20260928 08:30:59.298722 22111 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:59.298904 22111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:59.299232 22111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:59.302431 22111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/data/instance:
uuid: "5d00b51460aa4a2e93d8f6f353224a8a"
format_stamp: "Formatted at 2026-09-28 08:30:59 on dist-test-slave-qp2n"
server_key: "7ce7775d2980bc9d68053433dfcf4acf"
server_key_iv: "59dfe64c303b50439a92c6544a357fd0"
server_key_version: "encryptionkey@0"
I20260928 08:30:59.303341 22111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance:
uuid: "5d00b51460aa4a2e93d8f6f353224a8a"
format_stamp: "Formatted at 2026-09-28 08:30:59 on dist-test-slave-qp2n"
server_key: "7ce7775d2980bc9d68053433dfcf4acf"
server_key_iv: "59dfe64c303b50439a92c6544a357fd0"
server_key_version: "encryptionkey@0"
I20260928 08:30:59.309067 22111 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 08:30:59.313335 22126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:59.314764 22111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:30:59.314972 22111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "5d00b51460aa4a2e93d8f6f353224a8a"
format_stamp: "Formatted at 2026-09-28 08:30:59 on dist-test-slave-qp2n"
server_key: "7ce7775d2980bc9d68053433dfcf4acf"
server_key_iv: "59dfe64c303b50439a92c6544a357fd0"
server_key_version: "encryptionkey@0"
I20260928 08:30:59.315184 22111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:59.330436 22111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:59.334702 22111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:59.335021 22111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:59.336030 22111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:30:59.337445 22111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:59.337550 22111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:59.337703 22111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:59.337798 22111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:59.372386 22111 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:45771
I20260928 08:30:59.372476 22238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:45771 every 8 connection(s)
I20260928 08:30:59.373757 22111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:30:59.380349 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 22111
I20260928 08:30:59.380573 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance
I20260928 08:30:59.381253 15899 external_mini_cluster.cc:1428] Setting key 56cd5d7703aa96b7422f1e19f5e560e5
Sep 28 08:30:59 dist-test-slave-qp2n krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584259, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM
I20260928 08:30:59.394939 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:59.376419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:39429 (local address 127.15.134.254:44257)
0928 08:30:59.376963 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:59.376968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:59.377926 (+ 958us) server_negotiation.cc:407] Connection header received
0928 08:30:59.379737 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:59.379743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:59.379837 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:59.379956 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:59.381677 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:59.382585 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:59.384416 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:59.384920 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:59.385066 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:59.388983 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:59.388988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:59.388992 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:59.389027 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:59.391686 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:59.392487 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:59.392493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:59.392495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:59.392664 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:59.393132 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:59.393137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:59.393140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:59.393419 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:59.393576 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:59.394326 (+ 750us) server_negotiation.cc:299] Negotiation successful
0928 08:30:59.394567 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":287,"threads_started":1}
I20260928 08:30:59.395982 22242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:59.377048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.15.134.193:39429)
0928 08:30:59.377747 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:59.377791 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:59.379469 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:59.380195 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:59.380203 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:59.380861 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:59.381501 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:59.381522 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:59.382760 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:59.382766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:59.384234 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:59.384248 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:59.385164 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:59.385173 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:59.385441 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:59.386168 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:59.386191 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:59.388708 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:59.391882 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:59.391892 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:59.391919 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:59.392341 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:59.392786 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:59.392791 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:59.392795 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:59.393006 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:59.393552 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:59.393560 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:59.393932 (+ 372us) client_negotiation.cc:769] Sending connection context
0928 08:30:59.394209 (+ 277us) client_negotiation.cc:240] Negotiation successful
0928 08:30:59.394778 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":272,"threads_started":1}
I20260928 08:30:59.397718 22239 heartbeater.cc:347] Connected to a master server at 127.15.134.254:44257
I20260928 08:30:59.398096 22239 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:59.399088 22239 heartbeater.cc:511] Master 127.15.134.254:44257 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.15.134.194@KRBTEST.COM; defaulting to no policy
I20260928 08:30:59.401640 22050 ts_manager.cc:194] Registered new tserver with Master: 5d00b51460aa4a2e93d8f6f353224a8a (127.15.134.193:45771)
I20260928 08:30:59.403144 22050 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.134.193@KRBTEST.COM'} at 127.15.134.193:39429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.134.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.134.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.194.keytab.
Entry for principal oryx/127.15.134.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.194.keytab.
I20260928 08:30:59.430003 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:0
--local_ip_for_outbound_sockets=127.15.134.194
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:44257
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.194.keytab
--principal=oryx/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:46403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:30:59.779872 22245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:30:59.780179 22245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:30:59.780352 22245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:30:59.787992 22245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:30:59.788115 22245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:30:59.788307 22245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:30:59.798453 22245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:46403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.194.keytab
--principal=oryx/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:44257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:30:59.799823 22245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:30:59.801290 22245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:30:59.814652 22250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:59.814674 22253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:30:59.814608 22251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:30:59.815757 22245 server_base.cc:1034] running on GCE node
I20260928 08:30:59.816495 22245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:30:59.817597 22245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:30:59.818921 22245 hybrid_clock.cc:648] HybridClock initialized: now 1790584259818808 us; error 65 us; skew 500 ppm
I20260928 08:30:59.819264 22245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:30:59 dist-test-slave-qp2n krb5kdc[22005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584259, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:30:59.822652 22245 init.cc:377] Logged in from keytab as oryx/127.15.134.194@KRBTEST.COM (short username oryx)
I20260928 08:30:59.824205 22245 webserver.cc:460] Webserver started at http://127.15.134.194:46591/ using document root <none> and password file <none>
I20260928 08:30:59.824970 22245 fs_manager.cc:359] Metadata directory not provided
I20260928 08:30:59.825073 22245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:30:59.825350 22245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:30:59.828222 22245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/data/instance:
uuid: "b2602b66c20a4c90816bfe3ce31dde33"
format_stamp: "Formatted at 2026-09-28 08:30:59 on dist-test-slave-qp2n"
server_key: "95b38eae5a158da92611693abb5b2f7b"
server_key_iv: "b4958ec089ec6186cb90dcec8ac9bfd2"
server_key_version: "encryptionkey@0"
I20260928 08:30:59.829046 22245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance:
uuid: "b2602b66c20a4c90816bfe3ce31dde33"
format_stamp: "Formatted at 2026-09-28 08:30:59 on dist-test-slave-qp2n"
server_key: "95b38eae5a158da92611693abb5b2f7b"
server_key_iv: "b4958ec089ec6186cb90dcec8ac9bfd2"
server_key_version: "encryptionkey@0"
I20260928 08:30:59.834268 22245 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:30:59.838419 22260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:59.839883 22245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:30:59.840054 22245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "b2602b66c20a4c90816bfe3ce31dde33"
format_stamp: "Formatted at 2026-09-28 08:30:59 on dist-test-slave-qp2n"
server_key: "95b38eae5a158da92611693abb5b2f7b"
server_key_iv: "b4958ec089ec6186cb90dcec8ac9bfd2"
server_key_version: "encryptionkey@0"
I20260928 08:30:59.840277 22245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:30:59.859961 22245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:30:59.864555 22245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:30:59.864895 22245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:30:59.865929 22245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:30:59.867413 22245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:30:59.867497 22245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:59.867702 22245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:30:59.867760 22245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:30:59.903512 22245 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:35129
I20260928 08:30:59.903546 22372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:35129 every 8 connection(s)
I20260928 08:30:59.905022 22245 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:30:59.912397 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 22245
I20260928 08:30:59.912671 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance
I20260928 08:30:59.913157 15899 external_mini_cluster.cc:1428] Setting key bf99a484703fa7830c3b431091710551
Sep 28 08:30:59 dist-test-slave-qp2n krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584259, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM
I20260928 08:30:59.928913 22375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:59.907806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:32959 (local address 127.15.134.254:44257)
0928 08:30:59.908232 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0928 08:30:59.908236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:30:59.909394 (+ 1158us) server_negotiation.cc:407] Connection header received
0928 08:30:59.911305 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:30:59.911311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:30:59.911405 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:30:59.911528 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:30:59.913198 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:59.914024 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:59.916025 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:30:59.916505 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:30:59.916605 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:59.921736 (+ 5131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:30:59.921742 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:30:59.921745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:30:59.921802 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:30:59.925310 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:59.926161 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:59.926170 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:59.926176 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:59.926345 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:30:59.926770 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:30:59.926777 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:30:59.926783 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:30:59.927170 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:30:59.927342 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 08:30:59.928117 (+ 775us) server_negotiation.cc:299] Negotiation successful
0928 08:30:59.928379 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":203,"threads_started":1}
I20260928 08:30:59.929612 22376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:30:59.908373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.15.134.194:32959)
0928 08:30:59.909187 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0928 08:30:59.909225 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:30:59.911027 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:30:59.911779 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:30:59.911787 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:30:59.912317 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:30:59.913025 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:59.913043 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:59.914238 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:59.914249 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:59.915866 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:30:59.915884 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:30:59.916776 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:30:59.916787 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:30:59.917183 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:30:59.918059 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:30:59.918120 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:30:59.921444 (+ 3324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:30:59.925497 (+ 4053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:59.925514 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:59.925530 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:59.926009 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:59.926463 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:30:59.926466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:30:59.926468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:30:59.926643 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:30:59.927355 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:30:59.927366 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:30:59.927720 (+ 354us) client_negotiation.cc:769] Sending connection context
0928 08:30:59.927960 (+ 240us) client_negotiation.cc:240] Negotiation successful
0928 08:30:59.928362 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":345,"threads_started":1}
I20260928 08:30:59.931161 22373 heartbeater.cc:347] Connected to a master server at 127.15.134.254:44257
I20260928 08:30:59.931514 22373 heartbeater.cc:464] Registering TS with master...
I20260928 08:30:59.932461 22373 heartbeater.cc:511] Master 127.15.134.254:44257 requested a full tablet report, sending...
I20260928 08:30:59.934248 22050 ts_manager.cc:194] Registered new tserver with Master: b2602b66c20a4c90816bfe3ce31dde33 (127.15.134.194:35129)
I20260928 08:30:59.935105 22050 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.134.194@KRBTEST.COM'} at 127.15.134.194:32959
WARNING: no policy specified for oryx/127.15.134.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.134.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.134.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.195.keytab.
Entry for principal oryx/127.15.134.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.195.keytab.
I20260928 08:30:59.964170 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:0
--local_ip_for_outbound_sockets=127.15.134.195
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:44257
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.195.keytab
--principal=oryx/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:46403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:31:00.293883 22379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:00.294174 22379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:00.294271 22379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:00.302333 22379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:00.302460 22379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:00.302575 22379 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:31:00.312568 22379 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:46403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.195.keytab
--principal=oryx/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:44257
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:00.313962 22379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:00.315294 22379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:00.332897 22384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:00.332923 22387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:00.332948 22385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:00.333992 22379 server_base.cc:1034] running on GCE node
I20260928 08:31:00.334756 22379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:00.335803 22379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:00.337018 22379 hybrid_clock.cc:648] HybridClock initialized: now 1790584260336972 us; error 38 us; skew 500 ppm
I20260928 08:31:00.337375 22379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:31:00 dist-test-slave-qp2n krb5kdc[22005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584260, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:31:00.346179 22379 init.cc:377] Logged in from keytab as oryx/127.15.134.195@KRBTEST.COM (short username oryx)
I20260928 08:31:00.347865 22379 webserver.cc:460] Webserver started at http://127.15.134.195:35555/ using document root <none> and password file <none>
I20260928 08:31:00.348562 22379 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:00.348732 22379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:00.349009 22379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:31:00.352057 22379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/data/instance:
uuid: "cf9069423b204f8087751367c3f1cbc1"
format_stamp: "Formatted at 2026-09-28 08:31:00 on dist-test-slave-qp2n"
server_key: "5e473e327223c31f50704aabec7ae61e"
server_key_iv: "6b5c0b1b8148d993930c108a65cdd253"
server_key_version: "encryptionkey@0"
I20260928 08:31:00.352980 22379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance:
uuid: "cf9069423b204f8087751367c3f1cbc1"
format_stamp: "Formatted at 2026-09-28 08:31:00 on dist-test-slave-qp2n"
server_key: "5e473e327223c31f50704aabec7ae61e"
server_key_iv: "6b5c0b1b8148d993930c108a65cdd253"
server_key_version: "encryptionkey@0"
I20260928 08:31:00.358474 22379 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:31:00.361989 22394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:00.363420 22379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:31:00.363605 22379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "cf9069423b204f8087751367c3f1cbc1"
format_stamp: "Formatted at 2026-09-28 08:31:00 on dist-test-slave-qp2n"
server_key: "5e473e327223c31f50704aabec7ae61e"
server_key_iv: "6b5c0b1b8148d993930c108a65cdd253"
server_key_version: "encryptionkey@0"
I20260928 08:31:00.363827 22379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:00.389851 22379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:00.394058 22379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:00.394361 22379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:00.395368 22379 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:31:00.396836 22379 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:31:00.396956 22379 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:00.397109 22379 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:31:00.397183 22379 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:00.406127 22239 heartbeater.cc:503] Master 127.15.134.254:44257 was elected leader, sending a full tablet report...
I20260928 08:31:00.438876 22379 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:36669
I20260928 08:31:00.438918 22506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:36669 every 8 connection(s)
I20260928 08:31:00.440171 22379 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:31:00.445556 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 22379
I20260928 08:31:00.445730 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance
I20260928 08:31:00.446228 15899 external_mini_cluster.cc:1428] Setting key 746d14185809e9357a5a6081c650cc34
Sep 28 08:31:00 dist-test-slave-qp2n krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584260, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM
I20260928 08:31:00.460513 22509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:00.442851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:53707 (local address 127.15.134.254:44257)
0928 08:31:00.443352 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:00.443359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:00.444361 (+ 1002us) server_negotiation.cc:407] Connection header received
0928 08:31:00.446329 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:00.446336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:00.446428 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:00.446553 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:00.448210 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:00.449144 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:00.450716 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:00.451119 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:00.451257 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:00.454765 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:00.454770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:00.454773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:00.454809 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:00.457379 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:00.458121 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:00.458128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:00.458131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:00.458257 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:00.458676 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:00.458681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:00.458684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:00.458952 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:00.459113 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:00.459786 (+ 673us) server_negotiation.cc:299] Negotiation successful
0928 08:31:00.460005 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":221,"threads_started":1}
I20260928 08:31:00.461651 22510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:00.443443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.15.134.195:53707)
0928 08:31:00.444206 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:00.444244 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:00.446038 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:00.446763 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:00.446772 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:00.447367 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:00.448035 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:00.448053 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:00.449293 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:00.449301 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:00.450566 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:00.450578 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:00.451342 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:00.451349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:00.451589 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:00.452285 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:00.452309 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:00.454543 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:00.457534 (+ 2991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:00.457543 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:00.457562 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:00.457976 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:00.458374 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:00.458379 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:00.458383 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:00.458566 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:00.459080 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:00.459088 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:00.459431 (+ 343us) client_negotiation.cc:769] Sending connection context
0928 08:31:00.459702 (+ 271us) client_negotiation.cc:240] Negotiation successful
0928 08:31:00.460238 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":317,"threads_started":1}
I20260928 08:31:00.463037 22507 heartbeater.cc:347] Connected to a master server at 127.15.134.254:44257
I20260928 08:31:00.463368 22507 heartbeater.cc:464] Registering TS with master...
I20260928 08:31:00.464097 22507 heartbeater.cc:511] Master 127.15.134.254:44257 requested a full tablet report, sending...
I20260928 08:31:00.465576 22050 ts_manager.cc:194] Registered new tserver with Master: cf9069423b204f8087751367c3f1cbc1 (127.15.134.195:36669)
I20260928 08:31:00.466384 22050 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.134.195@KRBTEST.COM'} at 127.15.134.195:53707
I20260928 08:31:00.474623 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.15.134.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.134.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.134.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.253.keytab.
Entry for principal oryx/127.15.134.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.253.keytab.
I20260928 08:31:00.525081 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.253:39617
--webserver_interface=127.15.134.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.253.keytab
--principal=oryx/127.15.134.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:46403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:44257,127.15.134.253:39617
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:31:00.870170 22513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:00.870522 22513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:00.870730 22513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:00.878103 22513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:31:00.878255 22513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:00.878350 22513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:00.878454 22513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:31:00.878536 22513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:31:00.888454 22513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:46403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:44257,127.15.134.253:39617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.253.keytab
--principal=oryx/127.15.134.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.253:39617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.134.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
W20260928 08:31:00.888883 22513 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.15.134.254:44257,127.15.134.253:39617'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 08:31:00.890044 22513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:00.891547 22513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:00.903332 22518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:00.903416 22521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:00.903390 22519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:00.909040 22513 server_base.cc:1034] running on GCE node
I20260928 08:31:00.909821 22513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:00.911317 22513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:00.912649 22513 hybrid_clock.cc:648] HybridClock initialized: now 1790584260912510 us; error 44 us; skew 500 ppm
I20260928 08:31:00.912993 22513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:31:00 dist-test-slave-qp2n krb5kdc[22005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584260, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:31:00.916020 22513 init.cc:377] Logged in from keytab as oryx/127.15.134.253@KRBTEST.COM (short username oryx)
I20260928 08:31:00.917598 22513 webserver.cc:460] Webserver started at http://127.15.134.253:44625/ using document root <none> and password file <none>
I20260928 08:31:00.918234 22513 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:00.918344 22513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:00.918649 22513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:31:00.921628 22513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-1/data/instance:
uuid: "f424a72d7ff342a191e308a76893e581"
format_stamp: "Formatted at 2026-09-28 08:31:00 on dist-test-slave-qp2n"
server_key: "ef031e368c3d91c1f5ca9f9c2c837d0c"
server_key_iv: "0a3600607843cdcfa1fc5f063758de47"
server_key_version: "encryptionkey@0"
I20260928 08:31:00.922456 22513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-1/wal/instance:
uuid: "f424a72d7ff342a191e308a76893e581"
format_stamp: "Formatted at 2026-09-28 08:31:00 on dist-test-slave-qp2n"
server_key: "ef031e368c3d91c1f5ca9f9c2c837d0c"
server_key_iv: "0a3600607843cdcfa1fc5f063758de47"
server_key_version: "encryptionkey@0"
I20260928 08:31:00.927706 22513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:31:00.931334 22528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:00.932823 22513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:31:00.933002 22513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-1/wal
uuid: "f424a72d7ff342a191e308a76893e581"
format_stamp: "Formatted at 2026-09-28 08:31:00 on dist-test-slave-qp2n"
server_key: "ef031e368c3d91c1f5ca9f9c2c837d0c"
server_key_iv: "0a3600607843cdcfa1fc5f063758de47"
server_key_version: "encryptionkey@0"
I20260928 08:31:00.933243 22513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:00.938400 22373 heartbeater.cc:503] Master 127.15.134.254:44257 was elected leader, sending a full tablet report...
I20260928 08:31:00.948199 22513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:00.952891 22513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:00.953136 22513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:00.971875 22513 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.253:39617
I20260928 08:31:00.971892 22579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.253:39617 every 8 connection(s)
I20260928 08:31:00.973431 22513 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-1/data/info.pb
I20260928 08:31:00.975821 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 22513
I20260928 08:31:00.977731 22580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:00.987632 22580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:31:00 dist-test-slave-qp2n krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.134.253@KRBTEST.COM
Sep 28 08:31:00 dist-test-slave-qp2n krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584260, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.253@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM
I20260928 08:31:01.002899 22583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:00.978442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54226 (local address 127.15.134.253:39617)
0928 08:31:00.979395 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:00.979411 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:00.979899 (+ 488us) server_negotiation.cc:407] Connection header received
0928 08:31:00.982466 (+ 2567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:00.982497 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:00.983089 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:00.983577 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:00.985059 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:00.987015 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:00.988283 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:00.989290 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:00.989458 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:00.994453 (+ 4995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:00.994464 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:00.994483 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:00.994546 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:00.998715 (+ 4169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:00.999425 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:00.999435 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:00.999451 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:00.999603 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.000010 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.000019 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.000022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.000652 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:01.000966 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:01.001456 (+ 490us) server_negotiation.cc:299] Negotiation successful
0928 08:31:01.001894 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":416,"threads_started":1}
I20260928 08:31:01.006408 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:00.990576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51524 (local address 127.15.134.254:44257)
0928 08:31:00.991124 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:00.991131 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:00.991688 (+ 557us) server_negotiation.cc:407] Connection header received
0928 08:31:00.991904 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:00.991908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:00.991982 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:00.992070 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:00.993475 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:00.994273 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:00.996467 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:00.997006 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:00.997107 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.000841 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:01.000853 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:01.000856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:01.000893 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:01.003606 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.004459 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.004463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.004465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.004576 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.005038 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.005041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.005043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.005376 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:01.005540 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:01.006000 (+ 460us) server_negotiation.cc:299] Negotiation successful
0928 08:31:01.006169 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":252,"threads_started":1}
I20260928 08:31:01.006436 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:00.990624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.0.0.1:51524)
0928 08:31:00.991530 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:00.991578 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:00.991753 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:00.992183 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:00.992193 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:00.993066 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:00.993315 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:00.993324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:00.994387 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:00.994398 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:00.996306 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:00.996324 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:00.997206 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:00.997214 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:00.997400 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:00.998001 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:00.998031 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:01.000598 (+ 2567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:01.003746 (+ 3148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.003757 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.003764 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.004335 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.004739 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.004744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.004747 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.004932 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.005522 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:01.005531 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:01.005682 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 08:31:01.005908 (+ 226us) client_negotiation.cc:240] Negotiation successful
0928 08:31:01.006167 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":455,"threads_started":1}
I20260928 08:31:01.007927 22580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:31:01 dist-test-slave-qp2n krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584260, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.253@KRBTEST.COM for oryx/127.15.134.253@KRBTEST.COM
I20260928 08:31:01.019909 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.008670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:39617 (local address 127.0.0.1:54230)
0928 08:31:01.008845 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:01.008864 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:01.008958 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:01.009200 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:01.009203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:01.009531 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:01.009656 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.009690 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.010622 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.010627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.011544 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.011555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.012203 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.012208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.012430 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.012956 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:01.012971 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:01.014810 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:01.017508 (+ 2698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.017513 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.017517 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.017993 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.018375 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.018380 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.018384 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.018516 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.018977 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:01.018981 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:01.019119 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 08:31:01.019259 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 08:31:01.019538 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":55}
I20260928 08:31:01.019964 22583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.008595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54230 (local address 127.15.134.253:39617)
0928 08:31:01.008759 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:01.008766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:01.008932 (+ 166us) server_negotiation.cc:407] Connection header received
0928 08:31:01.009039 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:01.009043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:01.009094 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:01.009166 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:01.009805 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.010483 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.011687 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.012058 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.012193 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.014998 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:01.015003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:01.015009 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:01.015041 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:01.017379 (+ 2338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.018104 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.018112 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.018115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.018222 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.018616 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.018623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.018626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.018867 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:01.019012 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:01.019358 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 08:31:01.019676 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":123,"server-negotiator.queue_time_us":67}
I20260928 08:31:01.024725 22580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581: Bootstrap starting.
I20260928 08:31:01.027755 22580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:01.028852 22580 log.cc:824] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581: Log is configured to *not* fsync() on all Append() calls
I20260928 08:31:01.031386 22580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581: No bootstrap required, opened a new log
I20260928 08:31:01.038444 22580 raft_consensus.cc:348] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } }
I20260928 08:31:01.038793 22580 raft_consensus.cc:374] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:01.038908 22580 raft_consensus.cc:729] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f424a72d7ff342a191e308a76893e581, State: Initialized, Role: FOLLOWER
I20260928 08:31:01.039523 22580 consensus_queue.cc:260] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } }
I20260928 08:31:01.040418 22587 sys_catalog.cc:455] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } } }
I20260928 08:31:01.040822 22587 sys_catalog.cc:458] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:31:01.041544 22580 sys_catalog.cc:564] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 08:31:01.050714 22598 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:31:01.050918 22598 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:31:01.053401 22513 master_runner.cc:428] Detected that this master f424a72d7ff342a191e308a76893e581 is joining an existing cluster
I20260928 08:31:01.053505 22513 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.134.253:39617
I20260928 08:31:01.073936 22604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.057859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:39617 (local address 127.0.0.1:54246)
0928 08:31:01.058552 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:01.058572 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:01.058710 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:01.059085 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:01.059090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:01.059786 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:01.060194 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.060208 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.061852 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.061860 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.063584 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.063602 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.065118 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.065122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.065228 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.065897 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:01.065915 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:01.066765 (+ 850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:01.071003 (+ 4238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.071009 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.071013 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.071410 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.071795 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.071800 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.071804 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.071954 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.072778 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:01.072783 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:01.072924 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 08:31:01.073070 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 08:31:01.073411 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":391,"threads_started":1}
I20260928 08:31:01.075105 22583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.057690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54246 (local address 127.15.134.253:39617)
0928 08:31:01.057896 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:01.057903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:01.058670 (+ 767us) server_negotiation.cc:407] Connection header received
0928 08:31:01.058842 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:01.058849 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:01.058936 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:01.059042 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:01.060365 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.061597 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.063737 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.064647 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.064767 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.066970 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:01.066976 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:01.066981 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:01.067019 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:01.070851 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.071530 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.071538 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.071542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.071663 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.072067 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.072075 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.072079 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.072523 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:01.072707 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:01.074484 (+ 1777us) server_negotiation.cc:299] Negotiation successful
0928 08:31:01.074726 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":62}
I20260928 08:31:01.084470 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.061118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51528 (local address 127.15.134.254:44257)
0928 08:31:01.061307 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:01.061313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:01.068525 (+ 7212us) server_negotiation.cc:407] Connection header received
0928 08:31:01.068738 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:01.068746 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:01.068868 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:01.068943 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:01.070539 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.071661 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.073944 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.074661 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.074768 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.078038 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:01.078044 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:01.078047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:01.078083 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:01.081752 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.082430 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.082434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.082436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.082564 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.082959 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.082970 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.082973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.083249 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:01.083420 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:01.083870 (+ 450us) server_negotiation.cc:299] Negotiation successful
0928 08:31:01.084115 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:31:01.084966 22605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.061623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.0.0.1:51528)
0928 08:31:01.068357 (+ 6734us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:01.068385 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:01.068531 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:01.069195 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:01.069201 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:01.069980 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:01.070345 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.070358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.071890 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.071896 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.073731 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.073752 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.075123 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.075132 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.075505 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.076835 (+ 1330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:01.076862 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:01.077824 (+ 962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:01.081920 (+ 4096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.081926 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.081930 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.082287 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.082679 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.082685 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.082688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.082837 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.083373 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:01.083378 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:01.083601 (+ 223us) client_negotiation.cc:769] Sending connection context
0928 08:31:01.083738 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 08:31:01.084111 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6540,"mutex_wait_us":47,"thread_start_us":7012,"threads_started":1}
WARNING: no policy specified for oryx/127.15.134.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.134.252@KRBTEST.COM" created.
Sep 28 08:31:01 dist-test-slave-qp2n krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584258, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.254@KRBTEST.COM for oryx/127.15.134.253@KRBTEST.COM
I20260928 08:31:01.107554 22583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.091953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54262 (local address 127.15.134.253:39617)
0928 08:31:01.092187 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:01.092197 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:01.093308 (+ 1111us) server_negotiation.cc:407] Connection header received
0928 08:31:01.093692 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:01.093697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:01.093783 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:01.093890 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:01.095029 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.095751 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.097381 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.097923 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.097996 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.101911 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:01.101916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:01.101919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:01.101949 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:01.104731 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.105420 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.105425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.105429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.105557 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.105953 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.105957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.105961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.106241 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:01.106393 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:01.106935 (+ 542us) server_negotiation.cc:299] Negotiation successful
0928 08:31:01.107151 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 08:31:01.107607 22607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.091923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:39617 (local address 127.0.0.1:54262)
0928 08:31:01.093178 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:01.093215 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:01.093466 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:01.093946 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:01.093955 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:01.094647 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:01.094885 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.094895 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.095889 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.095895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.097185 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.097200 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.098194 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.098198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.098330 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.099004 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:01.099020 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:01.101716 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:01.104857 (+ 3141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.104869 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.104877 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.105290 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.105670 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.105676 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.105680 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.105839 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.106378 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:01.106387 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:01.106598 (+ 211us) client_negotiation.cc:769] Sending connection context
0928 08:31:01.106803 (+ 205us) client_negotiation.cc:240] Negotiation successful
0928 08:31:01.107151 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1023,"thread_start_us":860,"threads_started":1}
I20260928 08:31:01.109856 22050 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.134.253:39617: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f424a72d7ff342a191e308a76893e581"
member_type: NON_VOTER
last_known_addr {
host: "127.15.134.253"
port: 39617
}
attrs {
promote: true
}
}
dest_uuid: "7d3e3d1d841b4840ae24988248358275"
cas_config_opid_index: -1
I20260928 08:31:01.110991 22050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: NON_VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: true } }
I20260928 08:31:01.117110 22610 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f424a72d7ff342a191e308a76893e581 (127.15.134.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: NON_VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: true } } }
I20260928 08:31:01.117913 22555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:01.122431 22610 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.134.253:39617
I20260928 08:31:01.122534 22555 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [term 1 FOLLOWER]: Refusing update from remote peer 7d3e3d1d841b4840ae24988248358275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 08:31:01.123232 22611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7d3e3d1d841b4840ae24988248358275" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: NON_VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: true } } }
I20260928 08:31:01.123611 22611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [sys.catalog]: This master's current role is: LEADER
I20260928 08:31:01.123649 22513 master.cc:537] Master@127.15.134.253:39617 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.134.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.252.keytab.
Entry for principal oryx/127.15.134.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.252.keytab.
I20260928 08:31:01.124450 22610 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: NON_VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:31:01.125554 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.252:35423
--webserver_interface=127.15.134.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.252.keytab
--principal=oryx/127.15.134.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:46403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:44257,127.15.134.253:39617,127.15.134.252:35423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:31:01.136602 22034 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 08:31:01.138710 22034 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 -> Peer f424a72d7ff342a191e308a76893e581 (127.15.134.253:39617): Couldn't send request to peer f424a72d7ff342a191e308a76893e581. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 08:31:01.139740 22513 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:31:01.140131 22513 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:31:01.140334 22513 tablet_replica.cc:330] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581: stopping tablet replica
I20260928 08:31:01.157560 22513 master.cc:552] Master@127.15.134.253:39617 shutdown complete.
I20260928 08:31:01.157677 22513 master_runner.cc:305] Clearing existing system tablet
I20260928 08:31:01.158161 22513 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:01.158270 22513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:01.161686 22513 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 08:31:01.163475 22617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:01.163844 22513 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:01.164103 22513 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:31:01.164183 22513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-1/wal
uuid: "f424a72d7ff342a191e308a76893e581"
format_stamp: "Formatted at 2026-09-28 08:31:00 on dist-test-slave-qp2n"
server_key: "ef031e368c3d91c1f5ca9f9c2c837d0c"
server_key_iv: "0a3600607843cdcfa1fc5f063758de47"
server_key_version: "encryptionkey@0"
I20260928 08:31:01.169281 22513 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:31:01.176066 22513 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 08:31:01.176257 22513 log.cc:1196] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 08:31:01.176772 22513 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581: Deleting consensus metadata
I20260928 08:31:01.177058 22513 master_runner.cc:315] Copying system tablet from 127.15.134.254:44257
I20260928 08:31:01.179590 22513 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581: tablet copy: Beginning tablet copy session from remote peer at address 127.15.134.254:44257
I20260928 08:31:01.201403 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.180427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51536 (local address 127.15.134.254:44257)
0928 08:31:01.180603 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:01.180644 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:01.185324 (+ 4680us) server_negotiation.cc:407] Connection header received
0928 08:31:01.185546 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:01.185553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:01.185651 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:01.185785 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:01.186860 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.188138 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.190265 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.190947 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.191087 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.193685 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:01.193693 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:01.193698 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:01.193740 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:01.197517 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.198710 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.198716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.198720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.198887 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.199439 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.199444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.199448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.199866 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:01.200043 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:01.200716 (+ 673us) server_negotiation.cc:299] Negotiation successful
0928 08:31:01.200970 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 08:31:01.201720 22623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.180373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.0.0.1:51536)
0928 08:31:01.185147 (+ 4774us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:01.185170 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:01.185373 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:01.185832 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:01.185837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:01.186386 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:01.186678 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.186692 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.188341 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.188349 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.190073 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.190088 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.191149 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.191155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.191335 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.192139 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:01.192174 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:01.193449 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:01.197736 (+ 4287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.197746 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.197753 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.198526 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.199041 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.199048 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.199055 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.199285 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.200036 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:01.200044 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:01.200316 (+ 272us) client_negotiation.cc:769] Sending connection context
0928 08:31:01.200546 (+ 230us) client_negotiation.cc:240] Negotiation successful
0928 08:31:01.201077 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4581,"thread_start_us":4436,"threads_started":1}
I20260928 08:31:01.202785 22070 tablet_copy_service.cc:140] P 7d3e3d1d841b4840ae24988248358275: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f424a72d7ff342a191e308a76893e581 ({username='oryx', principal='oryx/127.15.134.253@KRBTEST.COM'} at 127.0.0.1:51536)
I20260928 08:31:01.203089 22070 tablet_copy_service.cc:161] P 7d3e3d1d841b4840ae24988248358275: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f424a72d7ff342a191e308a76893e581 at {username='oryx', principal='oryx/127.15.134.253@KRBTEST.COM'} at 127.0.0.1:51536: session id = f424a72d7ff342a191e308a76893e581-00000000000000000000000000000000
I20260928 08:31:01.207944 22070 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 08:31:01.210384 22513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 08:31:01.217317 22513 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581: tablet copy: Starting download of 0 data blocks...
I20260928 08:31:01.217639 22513 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581: tablet copy: Starting download of 1 WAL segments...
I20260928 08:31:01.220862 22513 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 08:31:01.226130 22070 tablet_copy_service.cc:342] P 7d3e3d1d841b4840ae24988248358275: Request end of tablet copy session f424a72d7ff342a191e308a76893e581-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.134.253@KRBTEST.COM'} at 127.0.0.1:51536
I20260928 08:31:01.226382 22070 tablet_copy_service.cc:433] P 7d3e3d1d841b4840ae24988248358275: ending tablet copy session f424a72d7ff342a191e308a76893e581-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f424a72d7ff342a191e308a76893e581
I20260928 08:31:01.237488 22513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:01.240231 22624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:01.246373 22625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:01.250919 22627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:01.253356 22513 server_base.cc:1034] running on GCE node
I20260928 08:31:01.253893 22513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:01.254693 22513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:01.256043 22513 hybrid_clock.cc:648] HybridClock initialized: now 1790584261255990 us; error 32 us; skew 500 ppm
I20260928 08:31:01.256392 22513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:31:01 dist-test-slave-qp2n krb5kdc[22005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584261, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:31:01.258428 22513 init.cc:377] Logged in from keytab as oryx/127.15.134.253@KRBTEST.COM (short username oryx)
I20260928 08:31:01.259785 22513 webserver.cc:460] Webserver started at http://127.15.134.253:44713/ using document root <none> and password file <none>
I20260928 08:31:01.260172 22513 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:01.260280 22513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:01.262916 22513 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260928 08:31:01.264783 22634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:01.265240 22513 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260928 08:31:01.265405 22513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-1/wal
uuid: "f424a72d7ff342a191e308a76893e581"
format_stamp: "Formatted at 2026-09-28 08:31:00 on dist-test-slave-qp2n"
server_key: "ef031e368c3d91c1f5ca9f9c2c837d0c"
server_key_iv: "0a3600607843cdcfa1fc5f063758de47"
server_key_version: "encryptionkey@0"
I20260928 08:31:01.265578 22513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:01.286376 22513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:01.290234 22513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:01.319461 22513 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.253:39617
I20260928 08:31:01.319480 22685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.253:39617 every 8 connection(s)
I20260928 08:31:01.320428 22513 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-1/data/info.pb
I20260928 08:31:01.328068 22686 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:31:01.330811 22686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581: Bootstrap starting.
I20260928 08:31:01.344311 22686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:31:01.344929 22686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581: Bootstrap complete.
I20260928 08:31:01.345866 22686 raft_consensus.cc:348] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: NON_VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: true } }
I20260928 08:31:01.346101 22686 raft_consensus.cc:729] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f424a72d7ff342a191e308a76893e581, State: Initialized, Role: LEARNER
I20260928 08:31:01.346350 22686 consensus_queue.cc:260] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: NON_VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: true } }
I20260928 08:31:01.347191 22689 sys_catalog.cc:455] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: NON_VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: true } } }
I20260928 08:31:01.347770 22686 sys_catalog.cc:564] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:31:01.347812 22689 sys_catalog.cc:458] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [sys.catalog]: This master's current role is: LEARNER
I20260928 08:31:01.361817 22700 catalog_manager.cc:1259] Loaded cluster ID: a09d68fb4c9f429dbd7b25f7f52fd6ec
I20260928 08:31:01.361932 22700 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581: loading cluster ID for follower catalog manager: success
I20260928 08:31:01.365499 22700 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581: acquiring CA information for follower catalog manager: success
I20260928 08:31:01.367904 22700 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:31:01.469270 22507 heartbeater.cc:503] Master 127.15.134.254:44257 was elected leader, sending a full tablet report...
W20260928 08:31:01.574492 22614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:01.574860 22614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:01.575013 22614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:01.582713 22614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:31:01.582897 22614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:01.583004 22614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:01.583118 22614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:31:01.583196 22614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:31:01.593412 22614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:46403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:44257,127.15.134.253:39617,127.15.134.252:35423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.252.keytab
--principal=oryx/127.15.134.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.252:35423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.134.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:01.594802 22614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:01.596120 22614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:01.608534 22706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:01.609095 22708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:01.611327 22705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:01.612586 22614 server_base.cc:1034] running on GCE node
I20260928 08:31:01.613449 22614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:01.614929 22614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:01.616175 22614 hybrid_clock.cc:648] HybridClock initialized: now 1790584261616111 us; error 84 us; skew 500 ppm
I20260928 08:31:01.616575 22614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:31:01 dist-test-slave-qp2n krb5kdc[22005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584261, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:31:01.620137 22614 init.cc:377] Logged in from keytab as oryx/127.15.134.252@KRBTEST.COM (short username oryx)
I20260928 08:31:01.621778 22614 webserver.cc:460] Webserver started at http://127.15.134.252:37595/ using document root <none> and password file <none>
I20260928 08:31:01.622629 22614 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:01.622769 22614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:01.623073 22614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:31:01.626127 22614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-2/data/instance:
uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5"
format_stamp: "Formatted at 2026-09-28 08:31:01 on dist-test-slave-qp2n"
server_key: "eff8858804b8bad443706896da9cf996"
server_key_iv: "3b85eac7972e9f19e768aed7283fa235"
server_key_version: "encryptionkey@0"
I20260928 08:31:01.627005 22614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-2/wal/instance:
uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5"
format_stamp: "Formatted at 2026-09-28 08:31:01 on dist-test-slave-qp2n"
server_key: "eff8858804b8bad443706896da9cf996"
server_key_iv: "3b85eac7972e9f19e768aed7283fa235"
server_key_version: "encryptionkey@0"
I20260928 08:31:01.632151 22614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260928 08:31:01.636327 22715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:01.637836 22614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:31:01.638047 22614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-2/wal
uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5"
format_stamp: "Formatted at 2026-09-28 08:31:01 on dist-test-slave-qp2n"
server_key: "eff8858804b8bad443706896da9cf996"
server_key_iv: "3b85eac7972e9f19e768aed7283fa235"
server_key_version: "encryptionkey@0"
I20260928 08:31:01.638247 22614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:01.677386 22614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:01.682560 22614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:01.682871 22614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:01.695943 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.687156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:39617 (local address 127.0.0.1:54278)
0928 08:31:01.688063 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:01.688078 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:01.688251 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:01.688690 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:01.688695 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:01.688723 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:01.688909 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.688918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.690678 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.690686 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.693182 (+ 2496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.693206 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.695127 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.695132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.695263 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.695314 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 08:31:01.695432 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 08:31:01.695581 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":708,"threads_started":1}
I20260928 08:31:01.695938 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.687372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54278 (local address 127.15.134.253:39617)
0928 08:31:01.687965 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:01.687972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:01.688175 (+ 203us) server_negotiation.cc:407] Connection header received
0928 08:31:01.688411 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:01.688415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:01.688495 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:01.688649 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:01.689053 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.690446 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.693435 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.694659 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.694848 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.695023 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:01.695458 (+ 435us) server_negotiation.cc:299] Negotiation successful
0928 08:31:01.695593 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":299,"threads_started":1}
I20260928 08:31:01.697556 22661 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260928 08:31:01.698292 22689 sys_catalog.cc:455] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d3e3d1d841b4840ae24988248358275. Latest consensus state: current_term: 1 leader_uuid: "7d3e3d1d841b4840ae24988248358275" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: NON_VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: true } } }
I20260928 08:31:01.698592 22689 sys_catalog.cc:458] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [sys.catalog]: This master's current role is: LEARNER
I20260928 08:31:01.706980 22614 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.252:35423
I20260928 08:31:01.707017 22769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.252:35423 every 8 connection(s)
I20260928 08:31:01.708458 22614 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-2/data/info.pb
I20260928 08:31:01.712491 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 22614
I20260928 08:31:01.712869 22770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:01.723157 22770 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:31:01 dist-test-slave-qp2n krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.134.252@KRBTEST.COM
Sep 28 08:31:01 dist-test-slave-qp2n krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584261, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.252@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM
I20260928 08:31:01.738989 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.715085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60664 (local address 127.15.134.252:35423)
0928 08:31:01.715940 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:01.715952 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:01.716001 (+ 49us) server_negotiation.cc:407] Connection header received
0928 08:31:01.718110 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:01.718134 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:01.718616 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:01.719054 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:01.720668 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.722134 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.723581 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.724407 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.724604 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.730225 (+ 5621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:01.730236 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:01.730250 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:01.730298 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:01.734152 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.735132 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.735143 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.735153 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.735451 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.735912 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.735919 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.735922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.736503 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:01.736853 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:01.737466 (+ 613us) server_negotiation.cc:299] Negotiation successful
0928 08:31:01.737874 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":357,"threads_started":1}
I20260928 08:31:01.742023 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.725875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51544 (local address 127.15.134.254:44257)
0928 08:31:01.727033 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:01.727038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:01.727052 (+ 14us) server_negotiation.cc:407] Connection header received
0928 08:31:01.727100 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:01.727104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:01.727160 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:01.727291 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:01.728364 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.729446 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.731305 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.731699 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.731796 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.735928 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:01.735933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:01.735937 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:01.735970 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:01.738704 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.739787 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.739792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.739795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.739954 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.740358 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.740363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.740366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.740740 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:01.740861 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:01.741496 (+ 635us) server_negotiation.cc:299] Negotiation successful
0928 08:31:01.741675 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1066,"thread_start_us":985,"threads_started":1}
I20260928 08:31:01.742137 22774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.725706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.0.0.1:51544)
0928 08:31:01.726479 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:01.726521 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:01.726670 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:01.727362 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:01.727368 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:01.727967 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:01.728205 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.728213 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.729586 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.729594 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.731058 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.731070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.731882 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.731888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.732044 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.732644 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:01.732673 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:01.735732 (+ 3059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:01.738839 (+ 3107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.738848 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.738855 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.739643 (+ 788us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.740103 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.740107 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.740109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.740247 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.740878 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:01.740886 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:01.741089 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 08:31:01.741349 (+ 260us) client_negotiation.cc:240] Negotiation successful
0928 08:31:01.741712 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":344,"threads_started":1}
I20260928 08:31:01.743928 22770 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:31:01 dist-test-slave-qp2n krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584261, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.252@KRBTEST.COM for oryx/127.15.134.253@KRBTEST.COM
I20260928 08:31:01.757189 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.744977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54282 (local address 127.15.134.253:39617)
0928 08:31:01.745145 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:01.745150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:01.745166 (+ 16us) server_negotiation.cc:407] Connection header received
0928 08:31:01.745215 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:01.745219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:01.745282 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:01.745358 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:01.745970 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.746728 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.747978 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.748389 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.748479 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.751625 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:01.751629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:01.751633 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:01.751664 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:01.754544 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.755393 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.755397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.755399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.755527 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.755948 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.755951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.755953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.756216 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:01.756334 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:01.756754 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 08:31:01.756946 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 08:31:01.757179 22774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.744735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:39617 (local address 127.0.0.1:54282)
0928 08:31:01.744965 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:01.744984 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:01.745093 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:01.745399 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:01.745402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:01.745706 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:01.745839 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.745846 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.746854 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.746858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.747862 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.747873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.748523 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.748527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.748725 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.749342 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:01.749360 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:01.751460 (+ 2100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:01.754721 (+ 3261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.754732 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.754735 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.755246 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.755646 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.755650 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.755652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.755854 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.756353 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:01.756357 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:01.756497 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 08:31:01.756707 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 08:31:01.756962 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 08:31:01.758630 22770 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.252" port: 35423 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:31:01 dist-test-slave-qp2n krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584261, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.252@KRBTEST.COM for oryx/127.15.134.252@KRBTEST.COM
I20260928 08:31:01.774268 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.759563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60680 (local address 127.15.134.252:35423)
0928 08:31:01.759802 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:01.759810 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:01.759831 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:31:01.759903 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:01.759911 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:01.760014 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:01.760136 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:01.760880 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.762264 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.764230 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.764850 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.765052 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.768451 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:01.768462 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:01.768465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:01.768509 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:01.771593 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.772232 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.772240 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.772243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.772358 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.772822 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.772830 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.772833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.773114 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:01.773266 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:01.773614 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 08:31:01.773966 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":144,"server-negotiator.queue_time_us":57}
I20260928 08:31:01.774271 22774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.759353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:35423 (local address 127.0.0.1:60680)
0928 08:31:01.759565 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:01.759583 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:01.759688 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:01.760152 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:01.760156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:01.760497 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:01.760735 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.760745 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.762442 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.762448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.764055 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.764069 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.765065 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.765070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.765298 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.765989 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:01.766006 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:01.768252 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:01.771739 (+ 3487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.771746 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.771750 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.772112 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.772478 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.772484 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.772487 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.772712 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.773239 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:01.773244 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:01.773398 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 08:31:01.773533 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 08:31:01.773818 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 08:31:01.781219 22770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5: Bootstrap starting.
I20260928 08:31:01.784905 22770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:01.786298 22770 log.cc:824] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5: Log is configured to *not* fsync() on all Append() calls
I20260928 08:31:01.790103 22770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5: No bootstrap required, opened a new log
I20260928 08:31:01.798527 22770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 35423 } }
I20260928 08:31:01.798951 22770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:01.799202 22770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e0c7d6a5dad44a88f0bb6262a2154f5, State: Initialized, Role: FOLLOWER
I20260928 08:31:01.799959 22770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 35423 } }
I20260928 08:31:01.801132 22777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 35423 } } }
I20260928 08:31:01.801510 22777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:31:01.802742 22770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 08:31:01.813762 22788 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:31:01.813933 22788 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:31:01.818643 22614 master_runner.cc:428] Detected that this master 8e0c7d6a5dad44a88f0bb6262a2154f5 is joining an existing cluster
I20260928 08:31:01.818794 22614 master_runner.cc:432] Initiating AddMaster RPC to add 127.15.134.252:35423
I20260928 08:31:01.846086 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 22111
I20260928 08:31:01.848007 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.823354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54294 (local address 127.15.134.253:39617)
0928 08:31:01.823551 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:01.823557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:01.829847 (+ 6290us) server_negotiation.cc:407] Connection header received
0928 08:31:01.830118 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:01.830124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:01.830202 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:01.830286 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:01.833278 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.834367 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.836475 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.837195 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.837291 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.840137 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:01.840144 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:01.840149 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:01.840187 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:01.843870 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.845545 (+ 1675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.845552 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.845556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.845717 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.846190 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.846197 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.846203 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.846569 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:01.846713 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:01.847388 (+ 675us) server_negotiation.cc:299] Negotiation successful
0928 08:31:01.847631 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 08:31:01.848399 22794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.823376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.0.0.1:51558)
0928 08:31:01.829051 (+ 5675us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:01.829079 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:01.829316 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:01.829955 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:01.829963 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:01.831047 (+ 1084us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:01.831920 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.831934 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.833477 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.833483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.835365 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.835383 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.836705 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.836712 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.837210 (+ 498us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.838018 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:01.838043 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:01.839260 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:01.843739 (+ 4479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.843744 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.843748 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.844690 (+ 942us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.846062 (+ 1372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.846070 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.846075 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.846262 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.847038 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:01.847042 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:01.847286 (+ 244us) client_negotiation.cc:769] Sending connection context
0928 08:31:01.847420 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 08:31:01.847761 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5459,"mutex_wait_us":318,"thread_start_us":5248,"threads_started":1}
I20260928 08:31:01.849038 22795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.823447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:39617 (local address 127.0.0.1:54294)
0928 08:31:01.829428 (+ 5981us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:01.829446 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:01.829949 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:01.830513 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:01.830519 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:01.832459 (+ 1940us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:01.833083 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.833101 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.834546 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.834551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.836317 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.836330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.837561 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.837566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.837796 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.838512 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:01.838536 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:01.839903 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:01.844042 (+ 4139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.844047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.844050 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.845417 (+ 1367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.845862 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.845867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.845871 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.846045 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.846794 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:01.846799 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:01.847006 (+ 207us) client_negotiation.cc:769] Sending connection context
0928 08:31:01.848206 (+ 1200us) client_negotiation.cc:240] Negotiation successful
0928 08:31:01.848569 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5854,"mutex_wait_us":5,"thread_start_us":5620,"threads_started":1}
I20260928 08:31:01.849704 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.823719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51558 (local address 127.15.134.254:44257)
0928 08:31:01.823964 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:01.823973 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:01.829186 (+ 5213us) server_negotiation.cc:407] Connection header received
0928 08:31:01.829485 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:01.829490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:01.829577 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:01.829655 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:01.832089 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.833255 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.835578 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.836297 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.836423 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.839513 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:01.839520 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:01.839525 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:01.839565 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:01.843577 (+ 4012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.845751 (+ 2174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.845756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.845759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.845918 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.846407 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.846413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.846417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.846806 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:01.846976 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:01.848878 (+ 1902us) server_negotiation.cc:299] Negotiation successful
0928 08:31:01.849196 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 08:31:01.853667 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.829557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60688 (local address 127.15.134.252:35423)
0928 08:31:01.829734 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:01.829740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:01.832125 (+ 2385us) server_negotiation.cc:407] Connection header received
0928 08:31:01.832336 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:01.832341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:01.832399 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:01.832461 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:01.833938 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.835043 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.837277 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.838029 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.838137 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.840813 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:01.840819 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:01.840824 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:01.840861 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:01.845121 (+ 4260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.850860 (+ 5739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.850868 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.850874 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.851039 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.851533 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.851540 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.851546 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.852004 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:01.852182 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:01.852854 (+ 672us) server_negotiation.cc:299] Negotiation successful
0928 08:31:01.853133 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":59}
I20260928 08:31:01.854998 22796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.829578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:35423 (local address 127.0.0.1:60688)
0928 08:31:01.832042 (+ 2464us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:01.832059 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:01.832209 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:01.832558 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:01.832562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:01.833373 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:01.833776 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.833787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.835271 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.835276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.837119 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.837137 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.838481 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.838488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.838688 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.839313 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:01.839334 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:01.840602 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:01.850022 (+ 9420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.850030 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.850037 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.850712 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.851209 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.851226 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.851232 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.851395 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.852282 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:01.852289 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:01.852502 (+ 213us) client_negotiation.cc:769] Sending connection context
0928 08:31:01.853908 (+ 1406us) client_negotiation.cc:240] Negotiation successful
0928 08:31:01.854301 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2309,"mutex_wait_us":27,"thread_start_us":2146,"threads_started":1}
Sep 28 08:31:01 dist-test-slave-qp2n krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584258, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.254@KRBTEST.COM for oryx/127.15.134.252@KRBTEST.COM
Sep 28 08:31:01 dist-test-slave-qp2n krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.134.193@KRBTEST.COM
I20260928 08:31:01.873838 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.856305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60690 (local address 127.15.134.252:35423)
0928 08:31:01.856467 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:01.856472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:01.856489 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:31:01.856574 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:01.856579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:01.856714 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:01.857705 (+ 991us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:01.857887 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.859001 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.860802 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.861505 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.861642 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.864933 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:01.864939 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:01.864944 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:01.864981 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:01.868936 (+ 3955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.869894 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.869899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.869903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.870041 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.871738 (+ 1697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.871742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.871744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.872191 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:01.872363 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:01.873062 (+ 699us) server_negotiation.cc:299] Negotiation successful
0928 08:31:01.873352 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 08:31:01.873910 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.855650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:35423 (local address 127.0.0.1:60690)
0928 08:31:01.855974 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:01.855993 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:01.856099 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:01.856885 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:01.856891 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:01.857373 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:01.857558 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.857570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.859160 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.859166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.860625 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.860643 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.861671 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.861677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.861864 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.862512 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:01.862540 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:01.864729 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:01.869118 (+ 4389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.869123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.869127 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.869772 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.870173 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.870178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.870182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.870488 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.872375 (+ 1887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:01.872381 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:01.872681 (+ 300us) client_negotiation.cc:769] Sending connection context
0928 08:31:01.872868 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 08:31:01.873287 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260928 08:31:01.876004 22050 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.15.134.252:35423: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5"
member_type: NON_VOTER
last_known_addr {
host: "127.15.134.252"
port: 35423
}
attrs {
promote: true
}
}
dest_uuid: "7d3e3d1d841b4840ae24988248358275"
cas_config_opid_index: 5
I20260928 08:31:01.877346 22050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: NON_VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: true } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: NON_VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: true } }
I20260928 08:31:01.879732 22722 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8e0c7d6a5dad44a88f0bb6262a2154f5 (127.15.134.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: NON_VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: true } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: NON_VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: true } } }
I20260928 08:31:01.880959 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.847853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41604 (local address 127.15.134.193:45771)
0928 08:31:01.858684 (+ 10831us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:01.858691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:01.858725 (+ 34us) server_negotiation.cc:407] Connection header received
0928 08:31:01.858776 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:01.858782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:01.859079 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:01.859509 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:01.861063 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.862467 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.864293 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.865120 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.865272 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.870757 (+ 5485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:01.870775 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:01.870790 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:01.870826 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:01.875188 (+ 4362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.876502 (+ 1314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.876511 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.876515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.876812 (+ 297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.877389 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.877394 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.877397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.877964 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:01.878380 (+ 416us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:01.879891 (+ 1511us) server_negotiation.cc:299] Negotiation successful
0928 08:31:01.880213 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10690,"thread_start_us":11370,"threads_started":1}
I20260928 08:31:01.882761 22234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:01.883453 22661 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [term 1 LEARNER]: Refusing update from remote peer 7d3e3d1d841b4840ae24988248358275: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 08:31:01.885535 22745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:01.892077 22798 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: NON_VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 08:31:01.893695 22722 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.15.134.252:35423
I20260928 08:31:01.899729 22799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7d3e3d1d841b4840ae24988248358275" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: NON_VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: true } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: NON_VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: true } } }
I20260928 08:31:01.901932 22799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [sys.catalog]: This master's current role is: LEADER
I20260928 08:31:01.909597 22661 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8e0c7d6a5dad44a88f0bb6262a2154f5 (127.15.134.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: NON_VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: true } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: NON_VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: true } } }
I20260928 08:31:01.915043 22689 sys_catalog.cc:455] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d3e3d1d841b4840ae24988248358275" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: NON_VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: true } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: NON_VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: true } } }
I20260928 08:31:01.915542 22689 sys_catalog.cc:458] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [sys.catalog]: This master's current role is: LEARNER
I20260928 08:31:01.915925 22799 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275: attempting to promote NON_VOTER f424a72d7ff342a191e308a76893e581 to VOTER
I20260928 08:31:01.917141 22614 master.cc:537] Master@127.15.134.252:35423 shutting down...
I20260928 08:31:01.917235 22745 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [term 1 FOLLOWER]: Refusing update from remote peer 7d3e3d1d841b4840ae24988248358275: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 08:31:01.917624 22799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: NON_VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: true } }
I20260928 08:31:01.921238 22661 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [term 1 LEARNER]: Refusing update from remote peer 7d3e3d1d841b4840ae24988248358275: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 08:31:01.921850 22798 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260928 08:31:01.923820 22034 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 -> Peer 8e0c7d6a5dad44a88f0bb6262a2154f5 (127.15.134.252:35423): Couldn't send request to peer 8e0c7d6a5dad44a88f0bb6262a2154f5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 08:31:01.925719 22799 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f424a72d7ff342a191e308a76893e581 (127.15.134.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: NON_VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: true } } }
I20260928 08:31:01.929563 22661 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f424a72d7ff342a191e308a76893e581 (127.15.134.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: NON_VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: true } } }
I20260928 08:31:01.930794 22798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7d3e3d1d841b4840ae24988248358275" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: NON_VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: true } } }
I20260928 08:31:01.931118 22798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [sys.catalog]: This master's current role is: LEADER
I20260928 08:31:01.931529 22798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d3e3d1d841b4840ae24988248358275" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: NON_VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: true } } }
I20260928 08:31:01.931780 22798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [sys.catalog]: This master's current role is: LEADER
I20260928 08:31:01.932505 22689 sys_catalog.cc:455] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d3e3d1d841b4840ae24988248358275" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: NON_VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: true } } }
I20260928 08:31:01.932796 22689 sys_catalog.cc:458] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:31:01.934005 22614 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:31:01.934358 22614 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:31:01.934512 22614 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5: stopping tablet replica
I20260928 08:31:01.940907 22614 master.cc:552] Master@127.15.134.252:35423 shutdown complete.
I20260928 08:31:01.941000 22614 master_runner.cc:305] Clearing existing system tablet
I20260928 08:31:01.941423 22614 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:01.941553 22614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:01.944430 22614 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260928 08:31:01.946215 22805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:01.946561 22614 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:01.946872 22614 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:31:01.946975 22614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-2/wal
uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5"
format_stamp: "Formatted at 2026-09-28 08:31:01 on dist-test-slave-qp2n"
server_key: "eff8858804b8bad443706896da9cf996"
server_key_iv: "3b85eac7972e9f19e768aed7283fa235"
server_key_version: "encryptionkey@0"
I20260928 08:31:01.950977 22614 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:31:01.956133 22614 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 08:31:01.956321 22614 log.cc:1196] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 08:31:01.956825 22614 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5: Deleting consensus metadata
I20260928 08:31:01.957348 22614 master_runner.cc:315] Copying system tablet from 127.15.134.254:44257
I20260928 08:31:01.960019 22614 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5: tablet copy: Beginning tablet copy session from remote peer at address 127.15.134.254:44257
I20260928 08:31:01.973982 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.960838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51566 (local address 127.15.134.254:44257)
0928 08:31:01.961107 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:01.961116 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:01.961617 (+ 501us) server_negotiation.cc:407] Connection header received
0928 08:31:01.961835 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:01.961841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:01.961921 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:01.962028 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:01.962799 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.963552 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.964869 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.965486 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.965629 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.968215 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:01.968221 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:01.968231 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:01.968261 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:01.971294 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.971939 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.971945 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.971948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.972083 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:01.972457 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:01.972464 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:01.972467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:01.972831 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:01.972985 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:01.973363 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 08:31:01.973586 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260928 08:31:01.974113 22811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:01.960828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.0.0.1:51566)
0928 08:31:01.961475 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:01.961495 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:01.961691 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:01.962051 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:01.962055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:01.962468 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:01.962640 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.962649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.963709 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.963715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.964731 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:01.964743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:01.965645 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:01.965651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:01.966138 (+ 487us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:01.966944 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:01.966969 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:01.967942 (+ 973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:01.971444 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.971449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.971453 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.971812 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.972196 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:01.972201 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:01.972205 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:01.972349 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:01.972944 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:01.972949 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:01.973118 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 08:31:01.973260 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 08:31:01.973574 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":436,"threads_started":1}
I20260928 08:31:01.974964 22070 tablet_copy_service.cc:140] P 7d3e3d1d841b4840ae24988248358275: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8e0c7d6a5dad44a88f0bb6262a2154f5 ({username='oryx', principal='oryx/127.15.134.252@KRBTEST.COM'} at 127.0.0.1:51566)
I20260928 08:31:01.975164 22070 tablet_copy_service.cc:161] P 7d3e3d1d841b4840ae24988248358275: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8e0c7d6a5dad44a88f0bb6262a2154f5 at {username='oryx', principal='oryx/127.15.134.252@KRBTEST.COM'} at 127.0.0.1:51566: session id = 8e0c7d6a5dad44a88f0bb6262a2154f5-00000000000000000000000000000000
I20260928 08:31:01.977789 22070 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 08:31:01.979408 22614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 08:31:01.986904 22614 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5: tablet copy: Starting download of 0 data blocks...
I20260928 08:31:01.987254 22614 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5: tablet copy: Starting download of 1 WAL segments...
I20260928 08:31:01.989735 22614 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 08:31:01.993522 22070 tablet_copy_service.cc:342] P 7d3e3d1d841b4840ae24988248358275: Request end of tablet copy session 8e0c7d6a5dad44a88f0bb6262a2154f5-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.15.134.252@KRBTEST.COM'} at 127.0.0.1:51566
I20260928 08:31:01.993683 22070 tablet_copy_service.cc:433] P 7d3e3d1d841b4840ae24988248358275: ending tablet copy session 8e0c7d6a5dad44a88f0bb6262a2154f5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8e0c7d6a5dad44a88f0bb6262a2154f5
I20260928 08:31:02.005975 22614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:02.010291 22812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:02.011142 22813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:02.018560 22815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:02.022009 22614 server_base.cc:1034] running on GCE node
I20260928 08:31:02.022401 22614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:02.023092 22614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:02.024310 22614 hybrid_clock.cc:648] HybridClock initialized: now 1790584262024261 us; error 30 us; skew 500 ppm
I20260928 08:31:02.024678 22614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:31:02 dist-test-slave-qp2n krb5kdc[22005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584262, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:31:02.027654 22614 init.cc:377] Logged in from keytab as oryx/127.15.134.252@KRBTEST.COM (short username oryx)
I20260928 08:31:02.034083 22614 webserver.cc:460] Webserver started at http://127.15.134.252:36177/ using document root <none> and password file <none>
I20260928 08:31:02.034664 22614 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:02.034930 22614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:02.038594 22614 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 08:31:02.040587 22822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:02.041112 22614 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260928 08:31:02.041290 22614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-2/wal
uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5"
format_stamp: "Formatted at 2026-09-28 08:31:01 on dist-test-slave-qp2n"
server_key: "eff8858804b8bad443706896da9cf996"
server_key_iv: "3b85eac7972e9f19e768aed7283fa235"
server_key_version: "encryptionkey@0"
I20260928 08:31:02.041600 22614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:02.043578 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 22111
I20260928 08:31:02.071429 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:45771
--local_ip_for_outbound_sockets=127.15.134.193
--tserver_master_addrs=127.15.134.254:44257
--webserver_port=35557
--webserver_interface=127.15.134.193
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.193.keytab
--principal=oryx/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:46403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.134.254:44257,127.15.134.253:39617
--tserver_master_addrs=127.15.134.254:44257,127.15.134.253:39617,127.15.134.252:35423 with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:31:02.080942 22614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:02.085760 22614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:02.111919 22614 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.252:35423
I20260928 08:31:02.111966 22874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.252:35423 every 8 connection(s)
I20260928 08:31:02.112982 22614 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/master-2/data/info.pb
I20260928 08:31:02.118212 22875 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:31:02.122138 22875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5: Bootstrap starting.
I20260928 08:31:02.140125 22875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:31:02.140666 22875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5: Bootstrap complete.
I20260928 08:31:02.141645 22875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: NON_VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: true } }
I20260928 08:31:02.141947 22875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8e0c7d6a5dad44a88f0bb6262a2154f5, State: Initialized, Role: LEARNER
I20260928 08:31:02.142186 22875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: NON_VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: true } }
I20260928 08:31:02.143260 22875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:31:02.142978 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: NON_VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: true } } }
I20260928 08:31:02.143596 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [sys.catalog]: This master's current role is: LEARNER
I20260928 08:31:02.162173 22889 catalog_manager.cc:1259] Loaded cluster ID: a09d68fb4c9f429dbd7b25f7f52fd6ec
I20260928 08:31:02.162303 22889 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5: loading cluster ID for follower catalog manager: success
I20260928 08:31:02.166041 22889 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5: acquiring CA information for follower catalog manager: success
I20260928 08:31:02.168542 22889 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:31:02.440186 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:02.430564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60702 (local address 127.15.134.252:35423)
0928 08:31:02.433632 (+ 3068us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:02.433637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:02.433655 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:31:02.433716 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:02.433720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:02.433794 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:02.433903 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:02.434488 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:02.435452 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:02.437883 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:02.439032 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:02.439219 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:02.439430 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:02.439693 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 08:31:02.439835 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2946,"thread_start_us":2819,"threads_started":1}
I20260928 08:31:02.440380 22890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:02.430371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:35423 (local address 127.0.0.1:60702)
0928 08:31:02.433024 (+ 2653us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:02.433049 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:02.433238 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:02.433978 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:02.433986 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:02.434018 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:02.434269 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:02.434281 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:02.435599 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:02.435606 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:02.437710 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:02.437732 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:02.439251 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:02.439258 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:02.439486 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:02.439535 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 08:31:02.439652 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 08:31:02.439832 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2492,"thread_start_us":2393,"threads_started":1}
I20260928 08:31:02.441883 22850 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 08:31:02.442472 22878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d3e3d1d841b4840ae24988248358275. Latest consensus state: current_term: 1 leader_uuid: "7d3e3d1d841b4840ae24988248358275" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: NON_VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: true } } }
I20260928 08:31:02.442802 22878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [sys.catalog]: This master's current role is: LEARNER
W20260928 08:31:02.477319 22827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:02.477680 22827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:02.477819 22827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:02.486059 22827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:02.486179 22827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:02.486299 22827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:31:02.496587 22827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:46403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.193.keytab
--principal=oryx/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:45771
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=35557
--tserver_master_addrs=127.15.134.254:44257,127.15.134.253:39617,127.15.134.252:35423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:02.498077 22827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:02.499617 22827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:02.515600 22896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:02.515686 22897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:02.515622 22899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:02.517372 22827 server_base.cc:1034] running on GCE node
I20260928 08:31:02.518218 22827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:02.519392 22827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:02.520696 22827 hybrid_clock.cc:648] HybridClock initialized: now 1790584262520577 us; error 64 us; skew 500 ppm
I20260928 08:31:02.521049 22827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:31:02 dist-test-slave-qp2n krb5kdc[22005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584262, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:31:02.524394 22827 init.cc:377] Logged in from keytab as oryx/127.15.134.193@KRBTEST.COM (short username oryx)
I20260928 08:31:02.525936 22827 webserver.cc:460] Webserver started at http://127.15.134.193:35557/ using document root <none> and password file <none>
I20260928 08:31:02.526575 22827 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:02.526690 22827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:02.533285 22827 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.006s
I20260928 08:31:02.536832 22906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:02.538431 22827 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 08:31:02.538609 22827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "5d00b51460aa4a2e93d8f6f353224a8a"
format_stamp: "Formatted at 2026-09-28 08:30:59 on dist-test-slave-qp2n"
server_key: "7ce7775d2980bc9d68053433dfcf4acf"
server_key_iv: "59dfe64c303b50439a92c6544a357fd0"
server_key_version: "encryptionkey@0"
I20260928 08:31:02.539649 22827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:02.562804 22827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:02.569216 22827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:02.569593 22827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:02.571056 22827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:31:02.573060 22827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:31:02.573261 22827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:02.573441 22827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:31:02.573532 22827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:02.618225 22827 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:45771
I20260928 08:31:02.618660 23018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:45771 every 8 connection(s)
I20260928 08:31:02.619951 22827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:31:02.622540 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 22827
I20260928 08:31:02.623070 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 22245
Sep 28 08:31:02 dist-test-slave-qp2n krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584262, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.252@KRBTEST.COM
I20260928 08:31:02.657966 22891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:02.627559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:50263 (local address 127.15.134.252:35423)
0928 08:31:02.627937 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:02.627947 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:02.628758 (+ 811us) server_negotiation.cc:407] Connection header received
0928 08:31:02.632904 (+ 4146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:02.632915 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:02.633033 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:02.633151 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:02.636345 (+ 3194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:02.637836 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:02.641808 (+ 3972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:02.642724 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:02.642852 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:02.648921 (+ 6069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:02.648931 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:02.648939 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:02.648993 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:02.653291 (+ 4298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:02.654563 (+ 1272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:02.654571 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:02.654577 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:02.654748 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:02.655445 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:02.655450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:02.655454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:02.655869 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:02.656001 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:02.657203 (+ 1202us) server_negotiation.cc:299] Negotiation successful
0928 08:31:02.657493 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260928 08:31:02.660789 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:02.625471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:35423 (local address 127.15.134.193:50263)
0928 08:31:02.628560 (+ 3089us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:02.628634 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:02.632335 (+ 3701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:02.633893 (+ 1558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:02.633910 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:02.634964 (+ 1054us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:02.636033 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:02.636065 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:02.638142 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:02.638152 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:02.641565 (+ 3413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:02.641588 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:02.643279 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:02.643291 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:02.643717 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:02.644696 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:02.644737 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:02.648595 (+ 3858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:02.653511 (+ 4916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:02.653525 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:02.653550 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:02.654405 (+ 855us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:02.654918 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:02.654924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:02.654927 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:02.655317 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:02.656094 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:02.656103 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:02.656669 (+ 566us) client_negotiation.cc:769] Sending connection context
0928 08:31:02.658474 (+ 1805us) client_negotiation.cc:240] Negotiation successful
0928 08:31:02.659045 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2694,"thread_start_us":5584,"threads_started":1}
I20260928 08:31:02.663159 23019 heartbeater.cc:347] Connected to a master server at 127.15.134.252:35423
I20260928 08:31:02.663578 23019 heartbeater.cc:464] Registering TS with master...
I20260928 08:31:02.664844 23019 heartbeater.cc:511] Master 127.15.134.252:35423 requested a full tablet report, sending...
I20260928 08:31:02.667865 22840 ts_manager.cc:194] Registered new tserver with Master: 5d00b51460aa4a2e93d8f6f353224a8a (127.15.134.193:45771)
Sep 28 08:31:02 dist-test-slave-qp2n krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584262, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM
Sep 28 08:31:02 dist-test-slave-qp2n krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.134.194@KRBTEST.COM
I20260928 08:31:02.683183 23025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:02.626823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.15.134.193:45621)
0928 08:31:02.650017 (+ 23194us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:02.650039 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:02.652913 (+ 2874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:02.656484 (+ 3571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:02.656489 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:02.657235 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:02.657694 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:02.657710 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:02.659207 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:02.659212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:02.661021 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:02.661034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:02.664037 (+ 3003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:02.664116 (+ 79us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:02.664332 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:02.666587 (+ 2255us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:02.666612 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:02.675080 (+ 8468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:02.679843 (+ 4763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:02.679849 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:02.679853 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:02.680426 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:02.680971 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:02.680975 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:02.680979 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:02.681144 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:02.681876 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:02.681884 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:02.682153 (+ 269us) client_negotiation.cc:769] Sending connection context
0928 08:31:02.682302 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 08:31:02.682643 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23012,"thread_start_us":24915,"threads_started":1}
I20260928 08:31:02.684238 23026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:02.628231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:45621 (local address 127.15.134.254:44257)
0928 08:31:02.655256 (+ 27025us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:02.655264 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:02.655287 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:31:02.655348 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:02.655356 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:02.655451 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:02.656316 (+ 865us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:02.657953 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:02.659046 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:02.661182 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:02.661803 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:02.661936 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:02.675405 (+ 13469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:02.675417 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:02.675424 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:02.675487 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:02.679697 (+ 4210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:02.680650 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:02.680657 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:02.680662 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:02.680828 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:02.681299 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:02.681304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:02.681307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:02.681727 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:02.683452 (+ 1725us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:02.683600 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 08:31:02.683835 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26885,"thread_start_us":26733,"threads_started":1}
Sep 28 08:31:02 dist-test-slave-qp2n krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584262, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.253@KRBTEST.COM
I20260928 08:31:02.685695 23020 heartbeater.cc:347] Connected to a master server at 127.15.134.254:44257
I20260928 08:31:02.685883 23020 heartbeater.cc:464] Registering TS with master...
I20260928 08:31:02.686432 23020 heartbeater.cc:511] Master 127.15.134.254:44257 requested a full tablet report, sending...
I20260928 08:31:02.687779 22050 ts_manager.cc:194] Re-registered known tserver with Master: 5d00b51460aa4a2e93d8f6f353224a8a (127.15.134.193:45771)
I20260928 08:31:02.689028 22050 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.134.193@KRBTEST.COM'} at 127.15.134.193:45621
I20260928 08:31:02.692831 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:02.625055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40890 (local address 127.15.134.194:35129)
0928 08:31:02.656173 (+ 31118us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:02.656182 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:02.662948 (+ 6766us) server_negotiation.cc:407] Connection header received
0928 08:31:02.663040 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:02.663073 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:02.663375 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:02.663549 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:02.670684 (+ 7135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:02.672895 (+ 2211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:02.676797 (+ 3902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:02.677796 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:02.677894 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:02.684263 (+ 6369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:02.684280 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:02.684300 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:02.684341 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:02.688865 (+ 4524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:02.689787 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:02.689795 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:02.689797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:02.689979 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:02.690448 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:02.690453 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:02.690456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:02.690971 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:02.691346 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:02.692009 (+ 663us) server_negotiation.cc:299] Negotiation successful
0928 08:31:02.692329 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30948,"thread_start_us":30680,"threads_started":1}
I20260928 08:31:02.693181 23024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:02.673490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:39617 (local address 127.15.134.193:39363)
0928 08:31:02.673872 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:02.673896 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:02.674030 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:02.676938 (+ 2908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:02.676945 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:02.677433 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:02.677623 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:02.677636 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:02.679361 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:02.679370 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:02.680967 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:02.680979 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:02.681948 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:02.681953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:02.682213 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:02.683022 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:02.683043 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:02.685992 (+ 2949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:02.690233 (+ 4241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:02.690238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:02.690242 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:02.690910 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:02.691329 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:02.691334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:02.691337 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:02.691556 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:02.692143 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:02.692158 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:02.692301 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 08:31:02.692464 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 08:31:02.692824 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189}
I20260928 08:31:02.694190 23028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:02.673466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:39363 (local address 127.15.134.253:39617)
0928 08:31:02.676484 (+ 3018us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:02.676489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:02.676518 (+ 29us) server_negotiation.cc:407] Connection header received
0928 08:31:02.676599 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:02.676606 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:02.676729 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:02.677800 (+ 1071us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:02.678045 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:02.679188 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:02.681142 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:02.681807 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:02.682497 (+ 690us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:02.686180 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:02.686187 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:02.686191 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:02.686234 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:02.690094 (+ 3860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:02.691044 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:02.691049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:02.691052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:02.691189 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:02.691688 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:02.691693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:02.691696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:02.692017 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:02.692178 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:02.692779 (+ 601us) server_negotiation.cc:299] Negotiation successful
0928 08:31:02.693932 (+ 1153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2917,"thread_start_us":2763,"threads_started":1}
I20260928 08:31:02.694288 22368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:02.695470 23022 heartbeater.cc:347] Connected to a master server at 127.15.134.253:39617
I20260928 08:31:02.695727 23022 heartbeater.cc:464] Registering TS with master...
I20260928 08:31:02.696382 23022 heartbeater.cc:511] Master 127.15.134.253:39617 requested a full tablet report, sending...
I20260928 08:31:02.698690 22651 ts_manager.cc:194] Registered new tserver with Master: 5d00b51460aa4a2e93d8f6f353224a8a (127.15.134.193:45771)
I20260928 08:31:02.782462 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 22245
I20260928 08:31:02.800180 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:35129
--local_ip_for_outbound_sockets=127.15.134.194
--tserver_master_addrs=127.15.134.254:44257
--webserver_port=46591
--webserver_interface=127.15.134.194
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.194.keytab
--principal=oryx/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:46403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.134.254:44257,127.15.134.253:39617
--tserver_master_addrs=127.15.134.254:44257,127.15.134.253:39617,127.15.134.252:35423 with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:31:02.967656 23031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d3e3d1d841b4840ae24988248358275" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: NON_VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: true } } }
I20260928 08:31:02.967999 23031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [sys.catalog]: This master's current role is: LEADER
I20260928 08:31:02.969058 22799 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275: attempting to promote NON_VOTER 8e0c7d6a5dad44a88f0bb6262a2154f5 to VOTER
I20260928 08:31:02.969698 22799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: false } }
I20260928 08:31:02.972034 22850 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [term 1 LEARNER]: Refusing update from remote peer 7d3e3d1d841b4840ae24988248358275: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 08:31:02.972754 22661 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [term 1 FOLLOWER]: Refusing update from remote peer 7d3e3d1d841b4840ae24988248358275: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 08:31:02.972733 22799 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 08:31:02.973414 23031 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 08:31:02.976392 23034 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8e0c7d6a5dad44a88f0bb6262a2154f5 (127.15.134.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: false } } }
I20260928 08:31:02.977351 22661 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8e0c7d6a5dad44a88f0bb6262a2154f5 (127.15.134.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: false } } }
I20260928 08:31:02.981104 22799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7d3e3d1d841b4840ae24988248358275" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: false } } }
I20260928 08:31:02.981194 23034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d3e3d1d841b4840ae24988248358275" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: false } } }
I20260928 08:31:02.981454 22799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [sys.catalog]: This master's current role is: LEADER
I20260928 08:31:02.981603 23034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [sys.catalog]: This master's current role is: LEADER
I20260928 08:31:02.981877 22799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d3e3d1d841b4840ae24988248358275" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: false } } }
I20260928 08:31:02.982209 22799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275 [sys.catalog]: This master's current role is: LEADER
I20260928 08:31:02.980381 22850 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8e0c7d6a5dad44a88f0bb6262a2154f5 (127.15.134.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: false } } }
I20260928 08:31:02.993791 23037 sys_catalog.cc:455] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d3e3d1d841b4840ae24988248358275" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: false } } }
I20260928 08:31:02.994180 23037 sys_catalog.cc:458] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:31:02.995010 23039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d3e3d1d841b4840ae24988248358275" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: false } } }
I20260928 08:31:02.995364 23039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 08:31:03.160813 23030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:03.161093 23030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:03.161265 23030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:03.169369 23030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:03.169497 23030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:03.169713 23030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:31:03.180111 23030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:46403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.194.keytab
--principal=oryx/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:35129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=46591
--tserver_master_addrs=127.15.134.254:44257,127.15.134.253:39617,127.15.134.252:35423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:03.181663 23030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:03.183200 23030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:03.197711 23047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:03.197734 23045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:03.197960 23044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:03.198822 23030 server_base.cc:1034] running on GCE node
I20260928 08:31:03.199549 23030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:03.200801 23030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:03.202076 23030 hybrid_clock.cc:648] HybridClock initialized: now 1790584263201977 us; error 85 us; skew 500 ppm
I20260928 08:31:03.202523 23030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:31:03 dist-test-slave-qp2n krb5kdc[22005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584263, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:31:03.206230 23030 init.cc:377] Logged in from keytab as oryx/127.15.134.194@KRBTEST.COM (short username oryx)
I20260928 08:31:03.207834 23030 webserver.cc:460] Webserver started at http://127.15.134.194:46591/ using document root <none> and password file <none>
I20260928 08:31:03.208593 23030 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:03.208896 23030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:03.215090 23030 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 08:31:03.218422 23054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:03.219755 23030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:31:03.219946 23030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "b2602b66c20a4c90816bfe3ce31dde33"
format_stamp: "Formatted at 2026-09-28 08:30:59 on dist-test-slave-qp2n"
server_key: "95b38eae5a158da92611693abb5b2f7b"
server_key_iv: "b4958ec089ec6186cb90dcec8ac9bfd2"
server_key_version: "encryptionkey@0"
I20260928 08:31:03.221027 23030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:03.238543 23030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:03.243113 23030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:03.243386 23030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:03.244647 23030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:31:03.246042 23030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:31:03.246127 23030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:03.246268 23030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:31:03.246372 23030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:03.280952 23030 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:35129
I20260928 08:31:03.281013 23166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:35129 every 8 connection(s)
I20260928 08:31:03.282588 23030 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:31:03.283732 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 23030
I20260928 08:31:03.284062 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 22379
Sep 28 08:31:03 dist-test-slave-qp2n krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584263, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.252@KRBTEST.COM
Sep 28 08:31:03 dist-test-slave-qp2n krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584263, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM
Sep 28 08:31:03 dist-test-slave-qp2n krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.134.195@KRBTEST.COM
I20260928 08:31:03.328140 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:03.285942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:37443 (local address 127.15.134.252:35423)
0928 08:31:03.305865 (+ 19923us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:03.305869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:03.305886 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:31:03.305938 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:03.305941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:03.306003 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:03.306135 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:03.308575 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.309836 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.312412 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.313329 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.314515 (+ 1186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:03.319126 (+ 4611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:03.319133 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:03.319138 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:03.319190 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:03.322538 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:03.323515 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:03.323520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:03.323525 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:03.323657 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:03.324242 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:03.324248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:03.324251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:03.324566 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:03.327432 (+ 2866us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:03.327559 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 08:31:03.327770 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19835,"thread_start_us":19754,"threads_started":1}
Sep 28 08:31:03 dist-test-slave-qp2n krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584263, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.253@KRBTEST.COM
I20260928 08:31:03.329710 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:03.286335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:35423 (local address 127.15.134.194:37443)
0928 08:31:03.297967 (+ 11632us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:03.298062 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:03.303954 (+ 5892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:03.306479 (+ 2525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:03.306493 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:03.307361 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:03.308346 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:03.308371 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.310011 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.310017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:03.312210 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:03.312224 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.313635 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.313645 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:03.314032 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:03.314876 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:03.314918 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:03.318882 (+ 3964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:03.322693 (+ 3811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:03.322703 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:03.322720 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:03.323377 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:03.323771 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:03.323776 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:03.323780 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:03.324094 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:03.324803 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:03.324812 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:03.325236 (+ 424us) client_negotiation.cc:769] Sending connection context
0928 08:31:03.325553 (+ 317us) client_negotiation.cc:240] Negotiation successful
0928 08:31:03.326191 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11125,"mutex_wait_us":58,"thread_start_us":21144,"threads_started":1}
I20260928 08:31:03.331877 23167 heartbeater.cc:347] Connected to a master server at 127.15.134.252:35423
I20260928 08:31:03.332221 23167 heartbeater.cc:464] Registering TS with master...
I20260928 08:31:03.333176 23167 heartbeater.cc:511] Master 127.15.134.252:35423 requested a full tablet report, sending...
I20260928 08:31:03.335069 22840 ts_manager.cc:194] Registered new tserver with Master: b2602b66c20a4c90816bfe3ce31dde33 (127.15.134.194:35129)
I20260928 08:31:03.335829 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:03.286188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:39083 (local address 127.15.134.254:44257)
0928 08:31:03.305589 (+ 19401us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:03.305594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:03.308047 (+ 2453us) server_negotiation.cc:407] Connection header received
0928 08:31:03.308325 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:03.308330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:03.308426 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:03.308510 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:03.309713 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.310960 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.317834 (+ 6874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.318580 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.318700 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:03.326609 (+ 7909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:03.326617 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:03.326623 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:03.326678 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:03.330656 (+ 3978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:03.332725 (+ 2069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:03.332730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:03.332734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:03.332904 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:03.334365 (+ 1461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:03.334370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:03.334374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:03.334654 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:03.334830 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:03.335215 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 08:31:03.335455 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19295,"thread_start_us":19228,"threads_started":1}
I20260928 08:31:03.336359 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:03.286335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.15.134.194:39083)
0928 08:31:03.307949 (+ 21614us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:03.307968 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:03.308154 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:03.308730 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:03.308737 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:03.309287 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:03.309551 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:03.309561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.311131 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.311136 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:03.316746 (+ 5610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:03.316757 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.319168 (+ 2411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.319177 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:03.319384 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:03.320675 (+ 1291us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:03.320700 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:03.326403 (+ 5703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:03.330911 (+ 4508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:03.330917 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:03.330921 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:03.331475 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:03.334077 (+ 2602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:03.334085 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:03.334091 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:03.334246 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:03.334802 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:03.334807 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:03.334966 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 08:31:03.335114 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 08:31:03.335420 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21456,"mutex_wait_us":2721,"thread_start_us":14091,"threads_started":1}
I20260928 08:31:03.337260 23168 heartbeater.cc:347] Connected to a master server at 127.15.134.254:44257
I20260928 08:31:03.337415 23168 heartbeater.cc:464] Registering TS with master...
I20260928 08:31:03.337832 23168 heartbeater.cc:511] Master 127.15.134.254:44257 requested a full tablet report, sending...
I20260928 08:31:03.338601 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:03.312180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:39617 (local address 127.15.134.194:51201)
0928 08:31:03.316384 (+ 4204us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:03.316407 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:03.316532 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:03.317020 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:03.317025 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:03.317550 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:03.317914 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:03.317929 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.322201 (+ 4272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.322209 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:03.323835 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:03.323851 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.325188 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.325198 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:03.325405 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:03.326629 (+ 1224us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:03.326654 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:03.329782 (+ 3128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:03.335356 (+ 5574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:03.335360 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:03.335363 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:03.335856 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:03.336355 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:03.336360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:03.336364 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:03.336826 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:03.337520 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:03.337525 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:03.337717 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 08:31:03.337887 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 08:31:03.338190 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4073,"thread_start_us":3659,"threads_started":1}
I20260928 08:31:03.338889 22050 ts_manager.cc:194] Re-registered known tserver with Master: b2602b66c20a4c90816bfe3ce31dde33 (127.15.134.194:35129)
I20260928 08:31:03.339350 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:03.285491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60350 (local address 127.15.134.195:36669)
0928 08:31:03.306780 (+ 21289us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:03.306789 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:03.307380 (+ 591us) server_negotiation.cc:407] Connection header received
0928 08:31:03.307543 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:03.307555 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:03.307833 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:03.308017 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:03.311539 (+ 3522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.313768 (+ 2229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.316061 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.317124 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.317280 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:03.325250 (+ 7970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:03.325267 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:03.325288 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:03.325333 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:03.333575 (+ 8242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:03.336090 (+ 2515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:03.336105 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:03.336108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:03.336309 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:03.336832 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:03.336836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:03.336840 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:03.337312 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:03.337576 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:03.338822 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0928 08:31:03.339061 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21109,"thread_start_us":20960,"threads_started":1}
I20260928 08:31:03.339839 22050 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.134.194@KRBTEST.COM'} at 127.15.134.194:39083
I20260928 08:31:03.340803 22502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:03.340852 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:03.308269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:51201 (local address 127.15.134.253:39617)
0928 08:31:03.315640 (+ 7371us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:03.315647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:03.316671 (+ 1024us) server_negotiation.cc:407] Connection header received
0928 08:31:03.316748 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:03.316752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:03.316848 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:03.316973 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:03.318125 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.320400 (+ 2275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.324022 (+ 3622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.324778 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.324908 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:03.330002 (+ 5094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:03.330009 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:03.330014 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:03.330069 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:03.335196 (+ 5127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:03.336039 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:03.336045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:03.336048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:03.336218 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:03.336967 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:03.336972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:03.336975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:03.337303 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:03.337439 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:03.338096 (+ 657us) server_negotiation.cc:299] Negotiation successful
0928 08:31:03.340434 (+ 2338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7252,"thread_start_us":7159,"threads_started":1}
I20260928 08:31:03.341784 23169 heartbeater.cc:347] Connected to a master server at 127.15.134.253:39617
I20260928 08:31:03.341910 23169 heartbeater.cc:464] Registering TS with master...
I20260928 08:31:03.342626 23169 heartbeater.cc:511] Master 127.15.134.253:39617 requested a full tablet report, sending...
I20260928 08:31:03.344328 22651 ts_manager.cc:194] Registered new tserver with Master: b2602b66c20a4c90816bfe3ce31dde33 (127.15.134.194:35129)
I20260928 08:31:03.434007 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 22379
I20260928 08:31:03.453354 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:36669
--local_ip_for_outbound_sockets=127.15.134.195
--tserver_master_addrs=127.15.134.254:44257
--webserver_port=35555
--webserver_interface=127.15.134.195
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.195.keytab
--principal=oryx/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:46403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.15.134.254:44257,127.15.134.253:39617
--tserver_master_addrs=127.15.134.254:44257,127.15.134.253:39617,127.15.134.252:35423 with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:31:03.692062 23020 heartbeater.cc:503] Master 127.15.134.254:44257 was elected leader, sending a full tablet report...
W20260928 08:31:03.803117 23180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:03.803449 23180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:03.803615 23180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:03.811357 23180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:03.811503 23180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:03.811725 23180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:31:03.822094 23180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:46403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.195.keytab
--principal=oryx/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:36669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=35555
--tserver_master_addrs=127.15.134.254:44257,127.15.134.253:39617,127.15.134.252:35423
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:03.823475 23180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:03.825029 23180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:03.838910 23186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:03.838909 23188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:03.839115 23185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:03.840693 23180 server_base.cc:1034] running on GCE node
I20260928 08:31:03.841360 23180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:03.842397 23180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:03.843631 23180 hybrid_clock.cc:648] HybridClock initialized: now 1790584263843581 us; error 50 us; skew 500 ppm
I20260928 08:31:03.844008 23180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:31:03 dist-test-slave-qp2n krb5kdc[22005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584263, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:31:03.847175 23180 init.cc:377] Logged in from keytab as oryx/127.15.134.195@KRBTEST.COM (short username oryx)
I20260928 08:31:03.848852 23180 webserver.cc:460] Webserver started at http://127.15.134.195:35555/ using document root <none> and password file <none>
I20260928 08:31:03.849606 23180 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:03.849749 23180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:03.855125 23180 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 08:31:03.858022 23195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:03.859440 23180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:31:03.859608 23180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "cf9069423b204f8087751367c3f1cbc1"
format_stamp: "Formatted at 2026-09-28 08:31:00 on dist-test-slave-qp2n"
server_key: "5e473e327223c31f50704aabec7ae61e"
server_key_iv: "6b5c0b1b8148d993930c108a65cdd253"
server_key_version: "encryptionkey@0"
I20260928 08:31:03.860694 23180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:03.885481 23180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:03.889611 23180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:03.889915 23180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:03.890978 23180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:31:03.892414 23180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:31:03.892499 23180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:03.892640 23180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:31:03.892715 23180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260928 08:31:03.928303 23180 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:36669
I20260928 08:31:03.928460 23307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:36669 every 8 connection(s)
I20260928 08:31:03.930051 23180 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:31:03.938947 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 23180
Sep 28 08:31:03 dist-test-slave-qp2n krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584263, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.252@KRBTEST.COM
Sep 28 08:31:03 dist-test-slave-qp2n krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584263, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM
Sep 28 08:31:03 dist-test-slave-qp2n krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584263, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.253@KRBTEST.COM
I20260928 08:31:03.973562 23315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:03.934065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:59455 (local address 127.15.134.254:44257)
0928 08:31:03.934830 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:03.934834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:03.937354 (+ 2520us) server_negotiation.cc:407] Connection header received
0928 08:31:03.938663 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:03.938669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:03.938759 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:03.938863 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:03.942022 (+ 3159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.943331 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.946687 (+ 3356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.947439 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.947566 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:03.960200 (+ 12634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:03.960212 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:03.960220 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:03.960281 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:03.965225 (+ 4944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:03.967507 (+ 2282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:03.967515 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:03.967520 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:03.967714 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:03.968368 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:03.968376 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:03.968380 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:03.968893 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:03.969070 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:03.972832 (+ 3762us) server_negotiation.cc:299] Negotiation successful
0928 08:31:03.973099 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":533,"threads_started":1}
I20260928 08:31:03.972575 23313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:03.933784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.15.134.195:59455)
0928 08:31:03.937229 (+ 3445us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:03.937250 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:03.938233 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:03.939324 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:03.939337 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:03.940740 (+ 1403us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:03.941764 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:03.941795 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.943546 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.943551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:03.946463 (+ 2912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:03.946491 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.947960 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.947968 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:03.948379 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:03.949418 (+ 1039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:03.949452 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:03.959801 (+ 10349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:03.965429 (+ 5628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:03.965442 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:03.965463 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:03.966307 (+ 844us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:03.967890 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:03.967896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:03.967902 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:03.968205 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:03.969176 (+ 971us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:03.969187 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:03.969723 (+ 536us) client_negotiation.cc:769] Sending connection context
0928 08:31:03.970088 (+ 365us) client_negotiation.cc:240] Negotiation successful
0928 08:31:03.970710 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3233,"mutex_wait_us":353,"spinlock_wait_cycles":93312,"thread_start_us":1014,"threads_started":1}
I20260928 08:31:03.975832 23309 heartbeater.cc:347] Connected to a master server at 127.15.134.254:44257
I20260928 08:31:03.976087 23317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:03.937517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:60679 (local address 127.15.134.252:35423)
0928 08:31:03.938156 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:03.938161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:03.938186 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:31:03.938666 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:03.938686 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:03.938800 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:03.938909 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:03.942045 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.943693 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.946655 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.947783 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.947965 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:03.966591 (+ 18626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:03.966600 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:03.966605 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:03.966659 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:03.970382 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:03.972887 (+ 2505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:03.972893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:03.972898 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:03.973062 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:03.973572 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:03.973581 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:03.973587 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:03.974568 (+ 981us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:03.974731 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:03.975312 (+ 581us) server_negotiation.cc:299] Negotiation successful
0928 08:31:03.975547 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":430,"threads_started":1}
I20260928 08:31:03.976246 23309 heartbeater.cc:464] Registering TS with master...
I20260928 08:31:03.977277 23309 heartbeater.cc:511] Master 127.15.134.254:44257 requested a full tablet report, sending...
I20260928 08:31:03.979329 22050 ts_manager.cc:194] Re-registered known tserver with Master: cf9069423b204f8087751367c3f1cbc1 (127.15.134.195:36669)
I20260928 08:31:03.980587 22050 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.134.195@KRBTEST.COM'} at 127.15.134.195:59455
I20260928 08:31:03.982095 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:03.933208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:51837 (local address 127.15.134.253:39617)
0928 08:31:03.933692 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:03.933699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:03.935508 (+ 1809us) server_negotiation.cc:407] Connection header received
0928 08:31:03.938572 (+ 3064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:03.938582 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:03.938698 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:03.938792 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:03.942021 (+ 3229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.943309 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.946980 (+ 3671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.947948 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.948057 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:03.964478 (+ 16421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:03.964489 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:03.964497 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:03.964585 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:03.971968 (+ 7383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:03.979044 (+ 7076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:03.979051 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:03.979054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:03.979278 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:03.979893 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:03.979900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:03.979904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:03.980316 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:03.980526 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:03.981323 (+ 797us) server_negotiation.cc:299] Negotiation successful
0928 08:31:03.981594 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":229,"threads_started":1}
I20260928 08:31:03.983428 23314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:03.933784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:39617 (local address 127.15.134.195:51837)
0928 08:31:03.935328 (+ 1544us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:03.935374 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:03.938233 (+ 2859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:03.939324 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:03.939337 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:03.940561 (+ 1224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:03.941764 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:03.941795 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.943621 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.943627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:03.946795 (+ 3168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:03.946815 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.948454 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.948461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:03.949069 (+ 608us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:03.951072 (+ 2003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:03.951106 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:03.963161 (+ 12055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:03.977493 (+ 14332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:03.977502 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:03.977507 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:03.978024 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:03.979526 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:03.979532 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:03.979535 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:03.979737 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:03.980800 (+ 1063us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:03.980805 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:03.980985 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 08:31:03.982596 (+ 1611us) client_negotiation.cc:240] Negotiation successful
0928 08:31:03.982939 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1290,"mutex_wait_us":321,"spinlock_wait_cycles":2432,"thread_start_us":2873,"threads_started":1}
I20260928 08:31:03.984232 23316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:03.937369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:35423 (local address 127.15.134.195:60679)
0928 08:31:03.937964 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:03.937983 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:03.938255 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:03.939323 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:03.939337 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:03.940770 (+ 1433us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:03.941789 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:03.941808 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.944082 (+ 2274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.944088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:03.946463 (+ 2375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:03.946491 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.948326 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.948335 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:03.948814 (+ 479us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:03.949623 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:03.949652 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:03.964980 (+ 15328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:03.972150 (+ 7170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:03.972155 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:03.972159 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:03.972721 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:03.973238 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:03.973243 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:03.973247 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:03.973430 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:03.974816 (+ 1386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:03.974820 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:03.974990 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 08:31:03.976645 (+ 1655us) client_negotiation.cc:240] Negotiation successful
0928 08:31:03.976994 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":181,"thread_start_us":269,"threads_started":1}
I20260928 08:31:03.985301 23308 heartbeater.cc:347] Connected to a master server at 127.15.134.252:35423
I20260928 08:31:03.985455 23308 heartbeater.cc:464] Registering TS with master...
I20260928 08:31:03.985567 23310 heartbeater.cc:347] Connected to a master server at 127.15.134.253:39617
I20260928 08:31:03.985680 23310 heartbeater.cc:464] Registering TS with master...
I20260928 08:31:03.985970 23308 heartbeater.cc:511] Master 127.15.134.252:35423 requested a full tablet report, sending...
I20260928 08:31:03.986050 23310 heartbeater.cc:511] Master 127.15.134.253:39617 requested a full tablet report, sending...
I20260928 08:31:03.987401 22840 ts_manager.cc:194] Registered new tserver with Master: cf9069423b204f8087751367c3f1cbc1 (127.15.134.195:36669)
I20260928 08:31:03.988273 22651 ts_manager.cc:194] Registered new tserver with Master: cf9069423b204f8087751367c3f1cbc1 (127.15.134.195:36669)
I20260928 08:31:04.004892 23315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:03.952818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51572 (local address 127.15.134.254:44257)
0928 08:31:03.973939 (+ 21121us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:03.973947 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:03.985212 (+ 11265us) server_negotiation.cc:407] Connection header received
0928 08:31:03.985535 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:03.985541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:03.985631 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:03.985741 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:03.989387 (+ 3646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.990347 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.992222 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.992865 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.992984 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:03.996960 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:03.996969 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:03.996975 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:03.997026 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:04.000600 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:04.001467 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:04.001476 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:04.001481 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:04.001658 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:04.002112 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:04.002118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:04.002123 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:04.002538 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:04.002829 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:04.004208 (+ 1379us) server_negotiation.cc:299] Negotiation successful
0928 08:31:04.004455 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21005,"thread_start_us":25998,"threads_started":1}
I20260928 08:31:04.004987 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:03.952265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54296 (local address 127.15.134.253:39617)
0928 08:31:03.982331 (+ 30066us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:03.982337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:03.987919 (+ 5582us) server_negotiation.cc:407] Connection header received
0928 08:31:03.988198 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:03.988204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:03.988314 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:03.988388 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:03.991504 (+ 3116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.992728 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.994555 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.994985 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.995124 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:03.998478 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:03.998484 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:03.998489 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:03.998531 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:04.001925 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:04.002624 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:04.002630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:04.002633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:04.002822 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:04.003268 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:04.003275 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:04.003280 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:04.003611 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:04.003877 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:04.004356 (+ 479us) server_negotiation.cc:299] Negotiation successful
0928 08:31:04.004571 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29962,"thread_start_us":31778,"threads_started":1}
I20260928 08:31:04.006206 23317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:03.987185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60712 (local address 127.15.134.252:35423)
0928 08:31:03.987430 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:03.987440 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:03.989132 (+ 1692us) server_negotiation.cc:407] Connection header received
0928 08:31:03.989487 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:03.989495 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:03.989582 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:03.989709 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:03.992149 (+ 2440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.993242 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.995260 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:03.995990 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:03.996099 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:03.999302 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:03.999310 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:03.999316 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:03.999358 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:04.002616 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:04.003326 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:04.003332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:04.003335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:04.003479 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:04.003903 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:04.003910 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:04.003912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:04.004258 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:04.004489 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:04.005056 (+ 567us) server_negotiation.cc:299] Negotiation successful
0928 08:31:04.005283 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 08:31:04.009212 22050 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:31:04.016335 22050 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:31:04.050369 23342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:04.039645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:35129 (local address 127.0.0.1:40898)
0928 08:31:04.040396 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:04.040421 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:04.040650 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:04.042267 (+ 1617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:04.042276 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:04.042313 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:04.042789 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:04.042803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.044856 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.044861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:04.047196 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:04.047210 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.049697 (+ 2487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.049704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:04.049871 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:04.049905 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 08:31:04.050003 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 08:31:04.050115 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":330,"threads_started":1}
I20260928 08:31:04.050519 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:04.039806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40898 (local address 127.15.134.194:35129)
0928 08:31:04.041141 (+ 1335us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:04.041155 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:04.041197 (+ 42us) server_negotiation.cc:407] Connection header received
0928 08:31:04.041335 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:04.041348 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:04.041733 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:04.041986 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:04.043076 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.044691 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.047907 (+ 3216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.049500 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.049689 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:04.049898 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:04.050048 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 08:31:04.050224 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1123,"thread_start_us":952,"threads_started":1}
I20260928 08:31:04.051592 23338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:04.036430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:36669 (local address 127.0.0.1:60358)
0928 08:31:04.037103 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:04.037119 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:04.037231 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:04.043098 (+ 5867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:04.043107 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:04.043142 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:04.043550 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:04.043563 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.045714 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.045719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:04.047967 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:04.047979 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.050825 (+ 2846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.050831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:04.051010 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:04.051044 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 08:31:04.051133 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 08:31:04.051284 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":409,"threads_started":1}
I20260928 08:31:04.051795 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:04.036808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60358 (local address 127.15.134.195:36669)
0928 08:31:04.037556 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:04.037568 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:04.037604 (+ 36us) server_negotiation.cc:407] Connection header received
0928 08:31:04.037730 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:04.037738 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:04.038070 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:04.038288 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:04.043781 (+ 5493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.045551 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.048123 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.049498 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.049701 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:04.049896 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:04.051277 (+ 1381us) server_negotiation.cc:299] Negotiation successful
0928 08:31:04.051449 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":407,"threads_started":1}
I20260928 08:31:04.052579 23340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:04.037806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:45771 (local address 127.0.0.1:41606)
0928 08:31:04.038390 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:04.038414 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:04.038566 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:04.044822 (+ 6256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:04.044826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:04.044857 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:04.045396 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:04.045406 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.047324 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.047329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:04.049476 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:04.049491 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.051726 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.051733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:04.051920 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:04.051953 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 08:31:04.052021 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 08:31:04.052178 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":24,"thread_start_us":272,"threads_started":1}
I20260928 08:31:04.053767 23341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:04.038047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41606 (local address 127.15.134.193:45771)
0928 08:31:04.044165 (+ 6118us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:04.044176 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:04.044204 (+ 28us) server_negotiation.cc:407] Connection header received
0928 08:31:04.044336 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:04.044345 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:04.044565 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:04.044852 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:04.045582 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.047172 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.049704 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.051484 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.053033 (+ 1549us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:04.053222 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:04.053297 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 08:31:04.053447 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5943,"thread_start_us":5821,"threads_started":1}
I20260928 08:31:04.055003 23102 tablet_service.cc:1467] Processing CreateTablet for tablet 3b5917f31b8e4565b86dffd81b8dc335 (DEFAULT_TABLE table=test-table [id=d0736cc0d63c4f19b290442b1bc44f8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:31:04.055975 23243 tablet_service.cc:1467] Processing CreateTablet for tablet 3b5917f31b8e4565b86dffd81b8dc335 (DEFAULT_TABLE table=test-table [id=d0736cc0d63c4f19b290442b1bc44f8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:31:04.056576 23102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b5917f31b8e4565b86dffd81b8dc335. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:04.057097 23243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b5917f31b8e4565b86dffd81b8dc335. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:04.057384 22954 tablet_service.cc:1467] Processing CreateTablet for tablet 3b5917f31b8e4565b86dffd81b8dc335 (DEFAULT_TABLE table=test-table [id=d0736cc0d63c4f19b290442b1bc44f8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:31:04.058653 22954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b5917f31b8e4565b86dffd81b8dc335. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:04.078683 23344 tablet_bootstrap.cc:492] T 3b5917f31b8e4565b86dffd81b8dc335 P cf9069423b204f8087751367c3f1cbc1: Bootstrap starting.
I20260928 08:31:04.079690 23345 tablet_bootstrap.cc:492] T 3b5917f31b8e4565b86dffd81b8dc335 P b2602b66c20a4c90816bfe3ce31dde33: Bootstrap starting.
I20260928 08:31:04.082444 23346 tablet_bootstrap.cc:492] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a: Bootstrap starting.
I20260928 08:31:04.082985 23344 tablet_bootstrap.cc:654] T 3b5917f31b8e4565b86dffd81b8dc335 P cf9069423b204f8087751367c3f1cbc1: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:04.084514 23345 tablet_bootstrap.cc:654] T 3b5917f31b8e4565b86dffd81b8dc335 P b2602b66c20a4c90816bfe3ce31dde33: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:04.084851 23344 log.cc:824] T 3b5917f31b8e4565b86dffd81b8dc335 P cf9069423b204f8087751367c3f1cbc1: Log is configured to *not* fsync() on all Append() calls
I20260928 08:31:04.085884 23345 log.cc:824] T 3b5917f31b8e4565b86dffd81b8dc335 P b2602b66c20a4c90816bfe3ce31dde33: Log is configured to *not* fsync() on all Append() calls
I20260928 08:31:04.087354 23346 tablet_bootstrap.cc:654] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:04.088802 23346 log.cc:824] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:31:04.088871 23344 tablet_bootstrap.cc:492] T 3b5917f31b8e4565b86dffd81b8dc335 P cf9069423b204f8087751367c3f1cbc1: No bootstrap required, opened a new log
I20260928 08:31:04.089141 23344 ts_tablet_manager.cc:1397] T 3b5917f31b8e4565b86dffd81b8dc335 P cf9069423b204f8087751367c3f1cbc1: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20260928 08:31:04.093739 23346 tablet_bootstrap.cc:492] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a: No bootstrap required, opened a new log
I20260928 08:31:04.094069 23345 tablet_bootstrap.cc:492] T 3b5917f31b8e4565b86dffd81b8dc335 P b2602b66c20a4c90816bfe3ce31dde33: No bootstrap required, opened a new log
I20260928 08:31:04.094059 23346 ts_tablet_manager.cc:1397] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260928 08:31:04.094337 23345 ts_tablet_manager.cc:1397] T 3b5917f31b8e4565b86dffd81b8dc335 P b2602b66c20a4c90816bfe3ce31dde33: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260928 08:31:04.100832 23344 raft_consensus.cc:348] T 3b5917f31b8e4565b86dffd81b8dc335 P cf9069423b204f8087751367c3f1cbc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9069423b204f8087751367c3f1cbc1" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 36669 } } peers { permanent_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 45771 } } peers { permanent_uuid: "b2602b66c20a4c90816bfe3ce31dde33" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35129 } }
I20260928 08:31:04.101336 23344 raft_consensus.cc:374] T 3b5917f31b8e4565b86dffd81b8dc335 P cf9069423b204f8087751367c3f1cbc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:04.101434 23344 raft_consensus.cc:729] T 3b5917f31b8e4565b86dffd81b8dc335 P cf9069423b204f8087751367c3f1cbc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf9069423b204f8087751367c3f1cbc1, State: Initialized, Role: FOLLOWER
I20260928 08:31:04.102156 23344 consensus_queue.cc:260] T 3b5917f31b8e4565b86dffd81b8dc335 P cf9069423b204f8087751367c3f1cbc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9069423b204f8087751367c3f1cbc1" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 36669 } } peers { permanent_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 45771 } } peers { permanent_uuid: "b2602b66c20a4c90816bfe3ce31dde33" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35129 } }
I20260928 08:31:04.102492 23345 raft_consensus.cc:348] T 3b5917f31b8e4565b86dffd81b8dc335 P b2602b66c20a4c90816bfe3ce31dde33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9069423b204f8087751367c3f1cbc1" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 36669 } } peers { permanent_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 45771 } } peers { permanent_uuid: "b2602b66c20a4c90816bfe3ce31dde33" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35129 } }
I20260928 08:31:04.102914 23345 raft_consensus.cc:374] T 3b5917f31b8e4565b86dffd81b8dc335 P b2602b66c20a4c90816bfe3ce31dde33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:04.103012 23345 raft_consensus.cc:729] T 3b5917f31b8e4565b86dffd81b8dc335 P b2602b66c20a4c90816bfe3ce31dde33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2602b66c20a4c90816bfe3ce31dde33, State: Initialized, Role: FOLLOWER
I20260928 08:31:04.103631 23345 consensus_queue.cc:260] T 3b5917f31b8e4565b86dffd81b8dc335 P b2602b66c20a4c90816bfe3ce31dde33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9069423b204f8087751367c3f1cbc1" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 36669 } } peers { permanent_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 45771 } } peers { permanent_uuid: "b2602b66c20a4c90816bfe3ce31dde33" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35129 } }
I20260928 08:31:04.104835 23309 heartbeater.cc:503] Master 127.15.134.254:44257 was elected leader, sending a full tablet report...
I20260928 08:31:04.105944 23346 raft_consensus.cc:348] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9069423b204f8087751367c3f1cbc1" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 36669 } } peers { permanent_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 45771 } } peers { permanent_uuid: "b2602b66c20a4c90816bfe3ce31dde33" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35129 } }
I20260928 08:31:04.106383 23346 raft_consensus.cc:374] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:04.106500 23346 raft_consensus.cc:729] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d00b51460aa4a2e93d8f6f353224a8a, State: Initialized, Role: FOLLOWER
I20260928 08:31:04.107147 23346 consensus_queue.cc:260] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9069423b204f8087751367c3f1cbc1" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 36669 } } peers { permanent_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 45771 } } peers { permanent_uuid: "b2602b66c20a4c90816bfe3ce31dde33" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35129 } }
I20260928 08:31:04.113473 23345 ts_tablet_manager.cc:1428] T 3b5917f31b8e4565b86dffd81b8dc335 P b2602b66c20a4c90816bfe3ce31dde33: Time spent starting tablet: real 0.019s user 0.009s sys 0.009s
I20260928 08:31:04.113600 23344 ts_tablet_manager.cc:1428] T 3b5917f31b8e4565b86dffd81b8dc335 P cf9069423b204f8087751367c3f1cbc1: Time spent starting tablet: real 0.024s user 0.011s sys 0.005s
I20260928 08:31:04.114933 23168 heartbeater.cc:503] Master 127.15.134.254:44257 was elected leader, sending a full tablet report...
I20260928 08:31:04.121340 23346 ts_tablet_manager.cc:1428] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a: Time spent starting tablet: real 0.027s user 0.007s sys 0.019s
W20260928 08:31:04.177551 23027 tablet.cc:2369] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:31:04.184924 23311 tablet.cc:2369] T 3b5917f31b8e4565b86dffd81b8dc335 P cf9069423b204f8087751367c3f1cbc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:31:04.279174 23352 raft_consensus.cc:482] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:31:04.279484 23352 raft_consensus.cc:504] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9069423b204f8087751367c3f1cbc1" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 36669 } } peers { permanent_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 45771 } } peers { permanent_uuid: "b2602b66c20a4c90816bfe3ce31dde33" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35129 } }
I20260928 08:31:04.282142 23352 leader_election.cc:288] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf9069423b204f8087751367c3f1cbc1 (127.15.134.195:36669), b2602b66c20a4c90816bfe3ce31dde33 (127.15.134.194:35129)
I20260928 08:31:04.291891 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:04.281910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:55673 (local address 127.15.134.194:35129)
0928 08:31:04.282169 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:04.282176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:04.283594 (+ 1418us) server_negotiation.cc:407] Connection header received
0928 08:31:04.283902 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:04.283911 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:04.283993 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:04.284057 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:04.285153 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.286046 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.289246 (+ 3200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.290886 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.291059 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:04.291136 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:04.291428 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 08:31:04.291572 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 08:31:04.291980 23353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:04.281917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:35129 (local address 127.15.134.193:55673)
0928 08:31:04.283490 (+ 1573us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:04.283509 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:04.283725 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:04.284295 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:04.284300 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:04.284326 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:04.284977 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:04.284998 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.286215 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.286221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:04.289073 (+ 2852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:04.289089 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.291110 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.291115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:04.291239 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:04.291267 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 08:31:04.291395 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 08:31:04.291570 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1405,"mutex_wait_us":137,"thread_start_us":1026,"threads_started":1}
I20260928 08:31:04.293210 23122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b5917f31b8e4565b86dffd81b8dc335" candidate_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2602b66c20a4c90816bfe3ce31dde33" is_pre_election: true
I20260928 08:31:04.293565 23354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:04.282793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:36669 (local address 127.15.134.193:49751)
0928 08:31:04.283930 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:04.283944 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:04.284095 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:04.284425 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:04.284429 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:04.284455 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:04.285027 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:04.285037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.286574 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.286579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:04.289073 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:04.289089 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.292920 (+ 3831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.292925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:04.293054 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:04.293079 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 08:31:04.293159 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 08:31:04.293266 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"mutex_wait_us":82,"thread_start_us":832,"threads_started":1}
I20260928 08:31:04.293675 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:04.281784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:49751 (local address 127.15.134.195:36669)
0928 08:31:04.281996 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:04.282003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:04.284041 (+ 2038us) server_negotiation.cc:407] Connection header received
0928 08:31:04.284246 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:04.284255 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:04.284313 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:04.284407 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:04.285206 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.286407 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.291230 (+ 4823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.292699 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.292894 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:04.292970 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:04.293232 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 08:31:04.293392 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 08:31:04.293874 23122 raft_consensus.cc:2454] T 3b5917f31b8e4565b86dffd81b8dc335 P b2602b66c20a4c90816bfe3ce31dde33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d00b51460aa4a2e93d8f6f353224a8a in term 0.
I20260928 08:31:04.294693 22909 leader_election.cc:302] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d00b51460aa4a2e93d8f6f353224a8a, b2602b66c20a4c90816bfe3ce31dde33; no voters:
I20260928 08:31:04.294673 23263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b5917f31b8e4565b86dffd81b8dc335" candidate_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf9069423b204f8087751367c3f1cbc1" is_pre_election: true
I20260928 08:31:04.295219 23352 raft_consensus.cc:2781] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:31:04.295387 23352 raft_consensus.cc:482] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:31:04.295315 23263 raft_consensus.cc:2454] T 3b5917f31b8e4565b86dffd81b8dc335 P cf9069423b204f8087751367c3f1cbc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d00b51460aa4a2e93d8f6f353224a8a in term 0.
I20260928 08:31:04.295485 23352 raft_consensus.cc:3037] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:04.299093 23352 raft_consensus.cc:504] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9069423b204f8087751367c3f1cbc1" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 36669 } } peers { permanent_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 45771 } } peers { permanent_uuid: "b2602b66c20a4c90816bfe3ce31dde33" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35129 } }
I20260928 08:31:04.299726 23352 leader_election.cc:288] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a [CANDIDATE]: Term 1 election: Requested vote from peers cf9069423b204f8087751367c3f1cbc1 (127.15.134.195:36669), b2602b66c20a4c90816bfe3ce31dde33 (127.15.134.194:35129)
I20260928 08:31:04.300181 23263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b5917f31b8e4565b86dffd81b8dc335" candidate_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf9069423b204f8087751367c3f1cbc1"
I20260928 08:31:04.300247 23122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b5917f31b8e4565b86dffd81b8dc335" candidate_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2602b66c20a4c90816bfe3ce31dde33"
I20260928 08:31:04.300428 23263 raft_consensus.cc:3037] T 3b5917f31b8e4565b86dffd81b8dc335 P cf9069423b204f8087751367c3f1cbc1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:04.300469 23122 raft_consensus.cc:3037] T 3b5917f31b8e4565b86dffd81b8dc335 P b2602b66c20a4c90816bfe3ce31dde33 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:04.302927 23263 raft_consensus.cc:2454] T 3b5917f31b8e4565b86dffd81b8dc335 P cf9069423b204f8087751367c3f1cbc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d00b51460aa4a2e93d8f6f353224a8a in term 1.
I20260928 08:31:04.303093 23122 raft_consensus.cc:2454] T 3b5917f31b8e4565b86dffd81b8dc335 P b2602b66c20a4c90816bfe3ce31dde33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d00b51460aa4a2e93d8f6f353224a8a in term 1.
I20260928 08:31:04.303438 22908 leader_election.cc:302] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d00b51460aa4a2e93d8f6f353224a8a, cf9069423b204f8087751367c3f1cbc1; no voters:
I20260928 08:31:04.303857 23352 raft_consensus.cc:2781] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:31:04.304471 23352 raft_consensus.cc:686] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a [term 1 LEADER]: Becoming Leader. State: Replica: 5d00b51460aa4a2e93d8f6f353224a8a, State: Running, Role: LEADER
I20260928 08:31:04.305107 23352 consensus_queue.cc:237] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9069423b204f8087751367c3f1cbc1" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 36669 } } peers { permanent_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 45771 } } peers { permanent_uuid: "b2602b66c20a4c90816bfe3ce31dde33" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35129 } }
W20260928 08:31:04.306181 23176 tablet.cc:2369] T 3b5917f31b8e4565b86dffd81b8dc335 P b2602b66c20a4c90816bfe3ce31dde33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:31:04.310667 22050 catalog_manager.cc:5495] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d00b51460aa4a2e93d8f6f353224a8a (127.15.134.193). New cstate: current_term: 1 leader_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9069423b204f8087751367c3f1cbc1" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 36669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 45771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2602b66c20a4c90816bfe3ce31dde33" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35129 } health_report { overall_health: UNKNOWN } } }
I20260928 08:31:04.347898 23341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:04.341871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41618 (local address 127.15.134.193:45771)
0928 08:31:04.342093 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:04.342099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:04.342275 (+ 176us) server_negotiation.cc:407] Connection header received
0928 08:31:04.342492 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:04.342496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:04.342575 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:04.342743 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:31:04.343124 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.344033 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.345508 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.346070 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.346227 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:04.346722 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:31:04.347266 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:31:04.347433 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:04.347535 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 08:31:04.347623 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:31:04.357003 23122 raft_consensus.cc:1261] T 3b5917f31b8e4565b86dffd81b8dc335 P b2602b66c20a4c90816bfe3ce31dde33 [term 1 FOLLOWER]: Refusing update from remote peer 5d00b51460aa4a2e93d8f6f353224a8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:31:04.357059 23263 raft_consensus.cc:1261] T 3b5917f31b8e4565b86dffd81b8dc335 P cf9069423b204f8087751367c3f1cbc1 [term 1 FOLLOWER]: Refusing update from remote peer 5d00b51460aa4a2e93d8f6f353224a8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:31:04.358224 23355 consensus_queue.cc:1035] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf9069423b204f8087751367c3f1cbc1" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 36669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:31:04.358618 23352 consensus_queue.cc:1035] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2602b66c20a4c90816bfe3ce31dde33" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:31:04.372006 23359 mvcc.cc:205] Tried to move back new op lower bound from 7334233146793836544 to 7334233146597658624. Current Snapshot: MvccSnapshot[applied={T|T < 7334233146793836544}]
I20260928 08:31:04.374672 23361 mvcc.cc:205] Tried to move back new op lower bound from 7334233146793836544 to 7334233146597658624. Current Snapshot: MvccSnapshot[applied={T|T < 7334233146793836544}]
I20260928 08:31:04.397166 22049 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:31:04.397828 22049 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:31:04.406629 22049 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7d3e3d1d841b4840ae24988248358275: Sending DeleteTablet for 3 replicas of tablet 3b5917f31b8e4565b86dffd81b8dc335
I20260928 08:31:04.407887 23243 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b5917f31b8e4565b86dffd81b8dc335 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:31:04 UTC) from {username='oryx', principal='oryx/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:60358
I20260928 08:31:04.408509 23102 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b5917f31b8e4565b86dffd81b8dc335 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:31:04 UTC) from {username='oryx', principal='oryx/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:40898
I20260928 08:31:04.408650 22954 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b5917f31b8e4565b86dffd81b8dc335 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:31:04 UTC) from {username='oryx', principal='oryx/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:41606
I20260928 08:31:04.409260 23368 tablet_replica.cc:330] T 3b5917f31b8e4565b86dffd81b8dc335 P cf9069423b204f8087751367c3f1cbc1: stopping tablet replica
I20260928 08:31:04.409768 23368 raft_consensus.cc:2229] T 3b5917f31b8e4565b86dffd81b8dc335 P cf9069423b204f8087751367c3f1cbc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:31:04.410535 23368 raft_consensus.cc:2258] T 3b5917f31b8e4565b86dffd81b8dc335 P cf9069423b204f8087751367c3f1cbc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:31:04.419745 23368 ts_tablet_manager.cc:1905] T 3b5917f31b8e4565b86dffd81b8dc335 P cf9069423b204f8087751367c3f1cbc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:31:04.426522 23370 tablet_replica.cc:330] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a: stopping tablet replica
I20260928 08:31:04.427131 23370 raft_consensus.cc:2229] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:31:04.427946 23370 raft_consensus.cc:2258] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:31:04.431957 23369 tablet_replica.cc:330] T 3b5917f31b8e4565b86dffd81b8dc335 P b2602b66c20a4c90816bfe3ce31dde33: stopping tablet replica
I20260928 08:31:04.432449 23369 raft_consensus.cc:2229] T 3b5917f31b8e4565b86dffd81b8dc335 P b2602b66c20a4c90816bfe3ce31dde33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:31:04.433113 23369 raft_consensus.cc:2258] T 3b5917f31b8e4565b86dffd81b8dc335 P b2602b66c20a4c90816bfe3ce31dde33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:31:04.435547 23369 ts_tablet_manager.cc:1905] T 3b5917f31b8e4565b86dffd81b8dc335 P b2602b66c20a4c90816bfe3ce31dde33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:31:04.440828 23370 ts_tablet_manager.cc:1905] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:31:04.442915 23368 ts_tablet_manager.cc:1918] T 3b5917f31b8e4565b86dffd81b8dc335 P cf9069423b204f8087751367c3f1cbc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:31:04.443142 23368 log.cc:1196] T 3b5917f31b8e4565b86dffd81b8dc335 P cf9069423b204f8087751367c3f1cbc1: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/3b5917f31b8e4565b86dffd81b8dc335
I20260928 08:31:04.443200 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 22018
I20260928 08:31:04.444585 22080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:04.444731 23368 ts_tablet_manager.cc:1939] T 3b5917f31b8e4565b86dffd81b8dc335 P cf9069423b204f8087751367c3f1cbc1: Deleting consensus metadata
I20260928 08:31:04.444231 23369 ts_tablet_manager.cc:1918] T 3b5917f31b8e4565b86dffd81b8dc335 P b2602b66c20a4c90816bfe3ce31dde33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:31:04.450523 23369 log.cc:1196] T 3b5917f31b8e4565b86dffd81b8dc335 P b2602b66c20a4c90816bfe3ce31dde33: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/3b5917f31b8e4565b86dffd81b8dc335
I20260928 08:31:04.451093 23369 ts_tablet_manager.cc:1939] T 3b5917f31b8e4565b86dffd81b8dc335 P b2602b66c20a4c90816bfe3ce31dde33: Deleting consensus metadata
I20260928 08:31:04.455163 23370 ts_tablet_manager.cc:1918] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:31:04.455494 23370 log.cc:1196] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/wal/wals/3b5917f31b8e4565b86dffd81b8dc335
I20260928 08:31:04.456301 23370 ts_tablet_manager.cc:1939] T 3b5917f31b8e4565b86dffd81b8dc335 P 5d00b51460aa4a2e93d8f6f353224a8a: Deleting consensus metadata
I20260928 08:31:04.464223 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:04.411489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54310 (local address 127.15.134.253:39617)
0928 08:31:04.411668 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:04.411675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:04.440330 (+ 28655us) server_negotiation.cc:407] Connection header received
0928 08:31:04.440563 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:04.440576 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:04.440730 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:04.440860 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:04.445854 (+ 4994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.447353 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.450422 (+ 3069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.451140 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.451292 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:04.455563 (+ 4271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:04.455569 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:04.455573 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:04.455643 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:04.461168 (+ 5525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:04.462074 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:04.462080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:04.462084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:04.462224 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:04.462605 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:04.462610 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:04.462613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:04.462914 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:04.463165 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:04.463515 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 08:31:04.463737 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 08:31:04.477474 23317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:04.440563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60726 (local address 127.15.134.252:35423)
0928 08:31:04.441143 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:04.441149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:04.464133 (+ 22984us) server_negotiation.cc:407] Connection header received
0928 08:31:04.464315 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:04.464321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:04.464398 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:04.464504 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:04.466151 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.466993 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.468253 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.468749 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.468880 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:04.471682 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:04.471689 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:04.471699 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:04.471756 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:04.474686 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:04.475362 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:04.475368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:04.475371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:04.475507 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:04.475886 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:04.475893 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:04.475896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:04.476177 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:04.476444 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:04.476765 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 08:31:04.477008 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436}
I20260928 08:31:04.634291 22035 catalog_manager.cc:4841] TS cf9069423b204f8087751367c3f1cbc1 (127.15.134.195:36669): tablet 3b5917f31b8e4565b86dffd81b8dc335 (table test-table [id=d0736cc0d63c4f19b290442b1bc44f8a]) successfully deleted
W20260928 08:31:04.634683 22035 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.15.134.195:36669, user_credentials={real_user=slave}} blocked reactor thread for 187613us
I20260928 08:31:04.635252 22036 catalog_manager.cc:4841] TS b2602b66c20a4c90816bfe3ce31dde33 (127.15.134.194:35129): tablet 3b5917f31b8e4565b86dffd81b8dc335 (table test-table [id=d0736cc0d63c4f19b290442b1bc44f8a]) successfully deleted
W20260928 08:31:04.635438 22036 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.15.134.194:35129, user_credentials={real_user=slave}} blocked reactor thread for 179820us
I20260928 08:31:04.635841 22036 catalog_manager.cc:4841] TS 5d00b51460aa4a2e93d8f6f353224a8a (127.15.134.193:45771): tablet 3b5917f31b8e4565b86dffd81b8dc335 (table test-table [id=d0736cc0d63c4f19b290442b1bc44f8a]) successfully deleted
I20260928 08:31:04.636348 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 22018
I20260928 08:31:04.688449 23317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:04.672000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60732 (local address 127.15.134.252:35423)
0928 08:31:04.672400 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:04.672407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:04.672429 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:31:04.672490 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:04.672495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:04.672574 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:04.672731 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:04.674705 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.675435 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.677375 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.678103 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.678233 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:04.682235 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:04.682246 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:04.682249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:04.682283 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:04.685264 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:04.685946 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:04.685952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:04.685956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:04.686145 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:04.686621 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:04.686626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:04.686629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:04.687017 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:04.687286 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:04.687653 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 08:31:04.687889 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 08:31:04.689347 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:04.670084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54318 (local address 127.15.134.253:39617)
0928 08:31:04.670350 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:04.670360 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:04.671518 (+ 1158us) server_negotiation.cc:407] Connection header received
0928 08:31:04.671595 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:04.671599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:04.671663 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:04.671778 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:04.674089 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.675643 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.677569 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:04.678004 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:04.678144 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:04.682216 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:04.682222 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:04.682226 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:04.682269 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:04.685865 (+ 3596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:04.686772 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:04.686785 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:04.686788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:04.686933 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:04.687333 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:04.687339 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:04.687342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:04.687723 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:04.687939 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:04.688424 (+ 485us) server_negotiation.cc:299] Negotiation successful
0928 08:31:04.689029 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 08:31:05.385402 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:05.384463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.15.134.193:49283)
0928 08:31:05.385029 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:05.385166 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":330,"threads_started":1}
W20260928 08:31:05.385908 23020 heartbeater.cc:644] Failed to heartbeat to 127.15.134.254:44257 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
I20260928 08:31:05.389930 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:05.389098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.15.134.195:52851)
0928 08:31:05.389591 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:05.389714 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":267,"threads_started":1}
W20260928 08:31:05.390305 23309 heartbeater.cc:644] Failed to heartbeat to 127.15.134.254:44257 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
I20260928 08:31:05.391921 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:05.390903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.15.134.194:47639)
0928 08:31:05.391584 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:05.391736 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":449,"threads_started":1}
W20260928 08:31:05.392258 23168 heartbeater.cc:644] Failed to heartbeat to 127.15.134.254:44257 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
I20260928 08:31:05.909286 23382 raft_consensus.cc:482] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7d3e3d1d841b4840ae24988248358275)
I20260928 08:31:05.909641 23382 raft_consensus.cc:504] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: false } }
I20260928 08:31:05.912499 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:05.911429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.0.0.1:52000)
0928 08:31:05.911971 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:05.912210 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":263,"threads_started":1}
W20260928 08:31:05.913160 22638 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
I20260928 08:31:05.914950 23383 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7d3e3d1d841b4840ae24988248358275)
I20260928 08:31:05.915158 23382 leader_election.cc:288] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7d3e3d1d841b4840ae24988248358275 (127.15.134.254:44257), 8e0c7d6a5dad44a88f0bb6262a2154f5 (127.15.134.252:35423)
I20260928 08:31:05.915232 23383 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: false } }
I20260928 08:31:05.916147 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:05.914984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.0.0.1:52006)
0928 08:31:05.915690 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:05.915951 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":586}
W20260928 08:31:05.916363 22638 leader_election.cc:334] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7d3e3d1d841b4840ae24988248358275 (127.15.134.254:44257): Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
I20260928 08:31:05.918236 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:05.916926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.0.0.1:52022)
0928 08:31:05.917664 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:05.917905 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":466,"threads_started":1}
W20260928 08:31:05.918557 22826 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
I20260928 08:31:05.920161 23383 leader_election.cc:288] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7d3e3d1d841b4840ae24988248358275 (127.15.134.254:44257), f424a72d7ff342a191e308a76893e581 (127.15.134.253:39617)
I20260928 08:31:05.922290 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:05.921881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.0.0.1:52032)
0928 08:31:05.922061 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:05.922150 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 08:31:05.922447 22826 leader_election.cc:334] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7d3e3d1d841b4840ae24988248358275 (127.15.134.254:44257): Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
I20260928 08:31:05.923475 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:05.912286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:35423 (local address 127.0.0.1:60748)
0928 08:31:05.912864 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:05.912899 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:05.913039 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:05.916478 (+ 3439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:05.916484 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:05.916513 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:05.916844 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:05.916856 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:05.918601 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:05.918607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:05.920833 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:05.920849 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:05.922715 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:05.922720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:05.922894 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:05.922919 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 08:31:05.923019 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 08:31:05.923172 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":3171,"threads_started":1}
I20260928 08:31:05.923601 23387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:05.915220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60748 (local address 127.15.134.252:35423)
0928 08:31:05.916094 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:05.916100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:05.916120 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:31:05.916209 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:05.916217 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:05.916297 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:05.916960 (+ 663us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:05.917155 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:05.918447 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:05.921045 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:05.922367 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:05.922531 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:05.922614 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:05.923091 (+ 477us) server_negotiation.cc:299] Negotiation successful
0928 08:31:05.923224 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":591,"threads_started":1}
I20260928 08:31:05.924269 22850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f424a72d7ff342a191e308a76893e581" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" is_pre_election: true
I20260928 08:31:05.924662 22850 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f424a72d7ff342a191e308a76893e581 in term 1.
I20260928 08:31:05.925225 22635 leader_election.cc:302] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0c7d6a5dad44a88f0bb6262a2154f5, f424a72d7ff342a191e308a76893e581; no voters: 7d3e3d1d841b4840ae24988248358275
I20260928 08:31:05.925626 23382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 08:31:05.925800 23382 raft_consensus.cc:482] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7d3e3d1d841b4840ae24988248358275)
I20260928 08:31:05.925937 23382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [term 1 FOLLOWER]: Advancing to term 2
I20260928 08:31:05.926818 23390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:05.918381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54334 (local address 127.15.134.253:39617)
0928 08:31:05.920275 (+ 1894us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:05.920281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:05.920301 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:31:05.920352 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:05.920358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:05.920416 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:05.920500 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:05.921458 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:05.922529 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:05.924519 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:05.925850 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:05.926024 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:05.926097 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:05.926448 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 08:31:05.926538 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1769,"thread_start_us":1585,"threads_started":1}
I20260928 08:31:05.926838 23389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:05.918046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:39617 (local address 127.0.0.1:54334)
0928 08:31:05.919565 (+ 1519us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:05.919584 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:05.919707 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:05.920576 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:05.920584 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:05.920647 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:05.921276 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:05.921288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:05.922648 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:05.922657 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:05.924281 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:05.924297 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:05.926028 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:05.926033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:05.926258 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:05.926307 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 08:31:05.926407 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 08:31:05.926528 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1400,"thread_start_us":1253,"threads_started":1}
I20260928 08:31:05.927563 22661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f424a72d7ff342a191e308a76893e581" is_pre_election: true
I20260928 08:31:05.928853 23382 raft_consensus.cc:504] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: false } }
I20260928 08:31:05.929327 22661 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e0c7d6a5dad44a88f0bb6262a2154f5 in current term 2: Already voted for candidate f424a72d7ff342a191e308a76893e581 in this term.
I20260928 08:31:05.929942 23382 leader_election.cc:288] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [CANDIDATE]: Term 2 election: Requested vote from peers 7d3e3d1d841b4840ae24988248358275 (127.15.134.254:44257), 8e0c7d6a5dad44a88f0bb6262a2154f5 (127.15.134.252:35423)
I20260928 08:31:05.929998 22850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f424a72d7ff342a191e308a76893e581" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5"
I20260928 08:31:05.930217 22850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [term 1 FOLLOWER]: Advancing to term 2
I20260928 08:31:05.930433 22823 leader_election.cc:302] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e0c7d6a5dad44a88f0bb6262a2154f5; no voters: 7d3e3d1d841b4840ae24988248358275, f424a72d7ff342a191e308a76893e581
I20260928 08:31:05.930546 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:05.929619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.0.0.1:52038)
0928 08:31:05.930304 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:05.930393 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573}
I20260928 08:31:05.931866 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:05.931470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.0.0.1:52052)
0928 08:31:05.931642 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:05.931744 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 08:31:05.932446 22638 leader_election.cc:334] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7d3e3d1d841b4840ae24988248358275 (127.15.134.254:44257): Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
I20260928 08:31:05.933189 22850 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f424a72d7ff342a191e308a76893e581 in term 2.
I20260928 08:31:05.933472 23383 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:31:05.933624 22635 leader_election.cc:302] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8e0c7d6a5dad44a88f0bb6262a2154f5, f424a72d7ff342a191e308a76893e581; no voters: 7d3e3d1d841b4840ae24988248358275
I20260928 08:31:05.934033 23382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 08:31:05.934573 23382 raft_consensus.cc:686] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [term 2 LEADER]: Becoming Leader. State: Replica: f424a72d7ff342a191e308a76893e581, State: Running, Role: LEADER
I20260928 08:31:05.935009 23382 consensus_queue.cc:237] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: false } }
I20260928 08:31:05.937077 23392 sys_catalog.cc:455] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f424a72d7ff342a191e308a76893e581. Latest consensus state: current_term: 2 leader_uuid: "f424a72d7ff342a191e308a76893e581" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: false } } }
I20260928 08:31:05.937551 23392 sys_catalog.cc:458] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [sys.catalog]: This master's current role is: LEADER
I20260928 08:31:05.938165 23394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:31:05.941681 23394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:31:05.942448 23394 catalog_manager.cc:1259] Loaded cluster ID: a09d68fb4c9f429dbd7b25f7f52fd6ec
I20260928 08:31:05.942556 23394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:31:05.943847 23394 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:31:05.944823 23394 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581: Loaded TSK: 0
I20260928 08:31:05.945650 23394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:31:05.993275 22650 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:31:06.000495 22650 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:31:06.004891 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.004347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.0.0.1:52076)
0928 08:31:06.004580 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:06.004699 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 08:31:06.004911 22850 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [term 2 FOLLOWER]: Refusing update from remote peer f424a72d7ff342a191e308a76893e581: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 08:31:06.005752 23392 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 08:31:06.006035 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.005669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.0.0.1:52090)
0928 08:31:06.005819 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:06.005898 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 08:31:06.006247 22638 consensus_peers.cc:487] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 -> Peer 7d3e3d1d841b4840ae24988248358275 (127.15.134.254:44257): Couldn't send request to peer 7d3e3d1d841b4840ae24988248358275. Status: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 08:31:06.009035 23383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f424a72d7ff342a191e308a76893e581. Latest consensus state: current_term: 2 leader_uuid: "f424a72d7ff342a191e308a76893e581" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: false } } }
I20260928 08:31:06.009428 23383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:31:06.011396 23396 mvcc.cc:205] Tried to move back new op lower bound from 7334233153548677120 to 7334233153273810944. Current Snapshot: MvccSnapshot[applied={T|T < 7334233153548677120}]
I20260928 08:31:06.011406 23392 sys_catalog.cc:455] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "f424a72d7ff342a191e308a76893e581" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: false } } }
I20260928 08:31:06.011919 23392 sys_catalog.cc:458] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581 [sys.catalog]: This master's current role is: LEADER
I20260928 08:31:06.012661 23383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f424a72d7ff342a191e308a76893e581" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7d3e3d1d841b4840ae24988248358275" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 44257 } } peers { permanent_uuid: "f424a72d7ff342a191e308a76893e581" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 39617 } attrs { promote: false } } peers { permanent_uuid: "8e0c7d6a5dad44a88f0bb6262a2154f5" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 35423 } attrs { promote: false } } }
I20260928 08:31:06.012923 23383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e0c7d6a5dad44a88f0bb6262a2154f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:31:06.029791 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.020537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:36669 (local address 127.0.0.1:60360)
0928 08:31:06.020767 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:06.020792 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:06.020920 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:06.021818 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:06.021823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:06.021852 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:06.022090 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:06.022105 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.024214 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.024228 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:06.026678 (+ 2450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:06.026702 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.028586 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.028595 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:06.028919 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:06.028986 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 08:31:06.029098 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 08:31:06.029307 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 08:31:06.032852 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.020878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60360 (local address 127.15.134.195:36669)
0928 08:31:06.021412 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:06.021421 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:06.021445 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:31:06.021551 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:06.021556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:06.021636 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:06.022259 (+ 623us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:06.022475 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.023970 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.026973 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.028367 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.030984 (+ 2617us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:06.031085 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:06.031142 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 08:31:06.032302 (+ 1160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":289,"threads_started":1}
I20260928 08:31:06.034560 23243 tablet_service.cc:1467] Processing CreateTablet for tablet fd7058748e3a4bd68544b715cafe2c82 (DEFAULT_TABLE table=test-table [id=47cb746187b949efb45008b204c6006c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:31:06.035068 23243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd7058748e3a4bd68544b715cafe2c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:06.039902 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.022037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40900 (local address 127.15.134.194:35129)
0928 08:31:06.033572 (+ 11535us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:06.033580 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:06.033604 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:31:06.033689 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:06.033697 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:06.033786 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:06.034367 (+ 581us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:06.034558 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.035489 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.037962 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.038786 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.039071 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:06.039164 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:06.039464 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 08:31:06.039605 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11383,"thread_start_us":11283,"threads_started":1}
I20260928 08:31:06.039899 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.021792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:35129 (local address 127.0.0.1:40900)
0928 08:31:06.030122 (+ 8330us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:06.030147 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:06.030304 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:06.033958 (+ 3654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:06.033964 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:06.033993 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:06.034215 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:06.034231 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.035664 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.035673 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:06.037171 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:06.037185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.039040 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.039047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:06.039288 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:06.039314 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 08:31:06.039437 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 08:31:06.039594 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8182,"thread_start_us":11711,"threads_started":1}
I20260928 08:31:06.040973 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.020946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:45771 (local address 127.0.0.1:41626)
0928 08:31:06.021073 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:06.021092 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:06.021224 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:06.034498 (+ 13274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:06.034503 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:06.034528 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:06.034813 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:06.034825 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.036353 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.036358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:06.038500 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:06.038517 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.040197 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.040203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:06.040365 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:06.040407 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 08:31:06.040474 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 08:31:06.040679 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":1}
I20260928 08:31:06.041364 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.021324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41626 (local address 127.15.134.193:45771)
0928 08:31:06.034159 (+ 12835us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:06.034167 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:06.034188 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:31:06.034270 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:06.034274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:06.034343 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:06.034980 (+ 637us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:06.035188 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.036185 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.038740 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.039960 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.040779 (+ 819us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:06.040866 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:06.040926 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 08:31:06.041067 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12666,"thread_start_us":12572,"threads_started":1}
I20260928 08:31:06.041465 23102 tablet_service.cc:1467] Processing CreateTablet for tablet fd7058748e3a4bd68544b715cafe2c82 (DEFAULT_TABLE table=test-table [id=47cb746187b949efb45008b204c6006c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:31:06.041947 23102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd7058748e3a4bd68544b715cafe2c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:06.042249 23405 tablet_bootstrap.cc:492] T fd7058748e3a4bd68544b715cafe2c82 P cf9069423b204f8087751367c3f1cbc1: Bootstrap starting.
I20260928 08:31:06.044283 23405 tablet_bootstrap.cc:654] T fd7058748e3a4bd68544b715cafe2c82 P cf9069423b204f8087751367c3f1cbc1: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:06.046295 22954 tablet_service.cc:1467] Processing CreateTablet for tablet fd7058748e3a4bd68544b715cafe2c82 (DEFAULT_TABLE table=test-table [id=47cb746187b949efb45008b204c6006c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:31:06.046752 22954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd7058748e3a4bd68544b715cafe2c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:06.047429 23405 tablet_bootstrap.cc:492] T fd7058748e3a4bd68544b715cafe2c82 P cf9069423b204f8087751367c3f1cbc1: No bootstrap required, opened a new log
I20260928 08:31:06.047606 23405 ts_tablet_manager.cc:1397] T fd7058748e3a4bd68544b715cafe2c82 P cf9069423b204f8087751367c3f1cbc1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 08:31:06.048317 23405 raft_consensus.cc:348] T fd7058748e3a4bd68544b715cafe2c82 P cf9069423b204f8087751367c3f1cbc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9069423b204f8087751367c3f1cbc1" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 36669 } } peers { permanent_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 45771 } } peers { permanent_uuid: "b2602b66c20a4c90816bfe3ce31dde33" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35129 } }
I20260928 08:31:06.048584 23405 raft_consensus.cc:374] T fd7058748e3a4bd68544b715cafe2c82 P cf9069423b204f8087751367c3f1cbc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:06.048728 23405 raft_consensus.cc:729] T fd7058748e3a4bd68544b715cafe2c82 P cf9069423b204f8087751367c3f1cbc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf9069423b204f8087751367c3f1cbc1, State: Initialized, Role: FOLLOWER
I20260928 08:31:06.048992 23405 consensus_queue.cc:260] T fd7058748e3a4bd68544b715cafe2c82 P cf9069423b204f8087751367c3f1cbc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9069423b204f8087751367c3f1cbc1" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 36669 } } peers { permanent_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 45771 } } peers { permanent_uuid: "b2602b66c20a4c90816bfe3ce31dde33" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35129 } }
I20260928 08:31:06.056772 23407 tablet_bootstrap.cc:492] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33: Bootstrap starting.
I20260928 08:31:06.057737 23405 ts_tablet_manager.cc:1428] T fd7058748e3a4bd68544b715cafe2c82 P cf9069423b204f8087751367c3f1cbc1: Time spent starting tablet: real 0.010s user 0.000s sys 0.011s
I20260928 08:31:06.061838 23407 tablet_bootstrap.cc:654] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:06.061820 23310 heartbeater.cc:503] Master 127.15.134.253:39617 was elected leader, sending a full tablet report...
I20260928 08:31:06.063469 23409 tablet_bootstrap.cc:492] T fd7058748e3a4bd68544b715cafe2c82 P 5d00b51460aa4a2e93d8f6f353224a8a: Bootstrap starting.
I20260928 08:31:06.065165 23409 tablet_bootstrap.cc:654] T fd7058748e3a4bd68544b715cafe2c82 P 5d00b51460aa4a2e93d8f6f353224a8a: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:06.067466 23407 tablet_bootstrap.cc:492] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33: No bootstrap required, opened a new log
I20260928 08:31:06.067664 23407 ts_tablet_manager.cc:1397] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260928 08:31:06.068789 23407 raft_consensus.cc:348] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9069423b204f8087751367c3f1cbc1" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 36669 } } peers { permanent_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 45771 } } peers { permanent_uuid: "b2602b66c20a4c90816bfe3ce31dde33" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35129 } }
I20260928 08:31:06.069110 23407 raft_consensus.cc:374] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:06.069243 23407 raft_consensus.cc:729] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2602b66c20a4c90816bfe3ce31dde33, State: Initialized, Role: FOLLOWER
I20260928 08:31:06.069561 23407 consensus_queue.cc:260] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9069423b204f8087751367c3f1cbc1" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 36669 } } peers { permanent_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 45771 } } peers { permanent_uuid: "b2602b66c20a4c90816bfe3ce31dde33" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35129 } }
I20260928 08:31:06.070547 23407 ts_tablet_manager.cc:1428] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:31:06.072846 23169 heartbeater.cc:503] Master 127.15.134.253:39617 was elected leader, sending a full tablet report...
I20260928 08:31:06.081868 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.070826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.15.134.194:39853)
0928 08:31:06.081217 (+ 10391us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:06.081473 (+ 256us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10197,"thread_start_us":10052,"threads_started":1}
I20260928 08:31:06.083046 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.058637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.15.134.195:40079)
0928 08:31:06.082621 (+ 23984us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:06.082758 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23859,"threads_started":1}
I20260928 08:31:06.083177 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.082749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.15.134.194:47119)
0928 08:31:06.082943 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:06.083036 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 08:31:06.084275 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.083769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.15.134.195:53735)
0928 08:31:06.083945 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:06.084028 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 08:31:06.084713 23409 tablet_bootstrap.cc:492] T fd7058748e3a4bd68544b715cafe2c82 P 5d00b51460aa4a2e93d8f6f353224a8a: No bootstrap required, opened a new log
I20260928 08:31:06.084853 23409 ts_tablet_manager.cc:1397] T fd7058748e3a4bd68544b715cafe2c82 P 5d00b51460aa4a2e93d8f6f353224a8a: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.008s
I20260928 08:31:06.085500 23409 raft_consensus.cc:348] T fd7058748e3a4bd68544b715cafe2c82 P 5d00b51460aa4a2e93d8f6f353224a8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9069423b204f8087751367c3f1cbc1" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 36669 } } peers { permanent_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 45771 } } peers { permanent_uuid: "b2602b66c20a4c90816bfe3ce31dde33" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35129 } }
I20260928 08:31:06.085711 23409 raft_consensus.cc:374] T fd7058748e3a4bd68544b715cafe2c82 P 5d00b51460aa4a2e93d8f6f353224a8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:06.085774 23409 raft_consensus.cc:729] T fd7058748e3a4bd68544b715cafe2c82 P 5d00b51460aa4a2e93d8f6f353224a8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d00b51460aa4a2e93d8f6f353224a8a, State: Initialized, Role: FOLLOWER
I20260928 08:31:06.085994 23409 consensus_queue.cc:260] T fd7058748e3a4bd68544b715cafe2c82 P 5d00b51460aa4a2e93d8f6f353224a8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9069423b204f8087751367c3f1cbc1" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 36669 } } peers { permanent_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 45771 } } peers { permanent_uuid: "b2602b66c20a4c90816bfe3ce31dde33" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35129 } }
I20260928 08:31:06.087322 23409 ts_tablet_manager.cc:1428] T fd7058748e3a4bd68544b715cafe2c82 P 5d00b51460aa4a2e93d8f6f353224a8a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:31:06.089129 23022 heartbeater.cc:503] Master 127.15.134.253:39617 was elected leader, sending a full tablet report...
I20260928 08:31:06.089586 23416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.087164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.15.134.193:57033)
0928 08:31:06.089063 (+ 1899us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:06.089276 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1774,"thread_start_us":1605,"threads_started":1}
I20260928 08:31:06.090723 23416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.090286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.15.134.193:46439)
0928 08:31:06.090501 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:06.090592 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260928 08:31:06.182315 23413 raft_consensus.cc:482] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:31:06.182538 23413 raft_consensus.cc:504] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9069423b204f8087751367c3f1cbc1" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 36669 } } peers { permanent_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 45771 } } peers { permanent_uuid: "b2602b66c20a4c90816bfe3ce31dde33" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35129 } }
I20260928 08:31:06.184158 23413 leader_election.cc:288] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf9069423b204f8087751367c3f1cbc1 (127.15.134.195:36669), 5d00b51460aa4a2e93d8f6f353224a8a (127.15.134.193:45771)
I20260928 08:31:06.190771 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.184633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:53789 (local address 127.15.134.193:45771)
0928 08:31:06.184972 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:06.184978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:06.185511 (+ 533us) server_negotiation.cc:407] Connection header received
0928 08:31:06.185770 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:06.185782 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:06.185840 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:06.185938 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:06.186429 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.187105 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.188965 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.189779 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.189971 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:06.190055 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:06.190355 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 08:31:06.190490 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260928 08:31:06.190985 23417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.184676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:45771 (local address 127.15.134.194:53789)
0928 08:31:06.185401 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:06.185415 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:06.185587 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:06.185953 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:06.185957 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:06.185978 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:06.186270 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:06.186278 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.187268 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.187276 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:06.188794 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:06.188807 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.189988 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.189994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:06.190185 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:06.190210 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 08:31:06.190329 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 08:31:06.190475 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"mutex_wait_us":4,"spinlock_wait_cycles":1920,"thread_start_us":439,"threads_started":1}
I20260928 08:31:06.191087 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.184584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:36669 (local address 127.15.134.194:34431)
0928 08:31:06.184877 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:06.184904 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:06.185110 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:06.185505 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:06.185511 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:06.185537 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:06.185856 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:06.185865 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.186935 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.186944 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:06.188814 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:06.188834 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.190183 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.190190 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:06.190366 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:06.190391 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 08:31:06.190507 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 08:31:06.190667 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":50,"spinlock_wait_cycles":2560}
I20260928 08:31:06.191534 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.184586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:34431 (local address 127.15.134.195:36669)
0928 08:31:06.184830 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:06.184837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:06.185033 (+ 196us) server_negotiation.cc:407] Connection header received
0928 08:31:06.185309 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:06.185314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:06.185378 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:06.185489 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:06.186010 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.186791 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.188993 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.189919 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.190128 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:06.190219 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:06.190534 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 08:31:06.190690 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260928 08:31:06.191915 22974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd7058748e3a4bd68544b715cafe2c82" candidate_uuid: "b2602b66c20a4c90816bfe3ce31dde33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" is_pre_election: true
I20260928 08:31:06.192233 23263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd7058748e3a4bd68544b715cafe2c82" candidate_uuid: "b2602b66c20a4c90816bfe3ce31dde33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf9069423b204f8087751367c3f1cbc1" is_pre_election: true
I20260928 08:31:06.192410 22974 raft_consensus.cc:2454] T fd7058748e3a4bd68544b715cafe2c82 P 5d00b51460aa4a2e93d8f6f353224a8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2602b66c20a4c90816bfe3ce31dde33 in term 0.
I20260928 08:31:06.192471 23263 raft_consensus.cc:2454] T fd7058748e3a4bd68544b715cafe2c82 P cf9069423b204f8087751367c3f1cbc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2602b66c20a4c90816bfe3ce31dde33 in term 0.
I20260928 08:31:06.193280 23056 leader_election.cc:302] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5d00b51460aa4a2e93d8f6f353224a8a, b2602b66c20a4c90816bfe3ce31dde33, cf9069423b204f8087751367c3f1cbc1; no voters:
I20260928 08:31:06.193843 23413 raft_consensus.cc:2781] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:31:06.193998 23413 raft_consensus.cc:482] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:31:06.194123 23413 raft_consensus.cc:3037] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:06.196426 23413 raft_consensus.cc:504] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9069423b204f8087751367c3f1cbc1" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 36669 } } peers { permanent_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 45771 } } peers { permanent_uuid: "b2602b66c20a4c90816bfe3ce31dde33" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35129 } }
I20260928 08:31:06.197288 23413 leader_election.cc:288] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33 [CANDIDATE]: Term 1 election: Requested vote from peers cf9069423b204f8087751367c3f1cbc1 (127.15.134.195:36669), 5d00b51460aa4a2e93d8f6f353224a8a (127.15.134.193:45771)
I20260928 08:31:06.197971 23263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd7058748e3a4bd68544b715cafe2c82" candidate_uuid: "b2602b66c20a4c90816bfe3ce31dde33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf9069423b204f8087751367c3f1cbc1"
I20260928 08:31:06.198021 22974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd7058748e3a4bd68544b715cafe2c82" candidate_uuid: "b2602b66c20a4c90816bfe3ce31dde33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d00b51460aa4a2e93d8f6f353224a8a"
I20260928 08:31:06.198205 23263 raft_consensus.cc:3037] T fd7058748e3a4bd68544b715cafe2c82 P cf9069423b204f8087751367c3f1cbc1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:06.198239 22974 raft_consensus.cc:3037] T fd7058748e3a4bd68544b715cafe2c82 P 5d00b51460aa4a2e93d8f6f353224a8a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:06.200536 23263 raft_consensus.cc:2454] T fd7058748e3a4bd68544b715cafe2c82 P cf9069423b204f8087751367c3f1cbc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2602b66c20a4c90816bfe3ce31dde33 in term 1.
I20260928 08:31:06.200950 22974 raft_consensus.cc:2454] T fd7058748e3a4bd68544b715cafe2c82 P 5d00b51460aa4a2e93d8f6f353224a8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2602b66c20a4c90816bfe3ce31dde33 in term 1.
I20260928 08:31:06.201149 23056 leader_election.cc:302] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2602b66c20a4c90816bfe3ce31dde33, cf9069423b204f8087751367c3f1cbc1; no voters:
I20260928 08:31:06.201511 23413 raft_consensus.cc:2781] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:31:06.201978 23413 raft_consensus.cc:686] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33 [term 1 LEADER]: Becoming Leader. State: Replica: b2602b66c20a4c90816bfe3ce31dde33, State: Running, Role: LEADER
I20260928 08:31:06.202455 23413 consensus_queue.cc:237] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9069423b204f8087751367c3f1cbc1" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 36669 } } peers { permanent_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 45771 } } peers { permanent_uuid: "b2602b66c20a4c90816bfe3ce31dde33" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35129 } }
I20260928 08:31:06.203126 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.202634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.15.134.194:54715)
0928 08:31:06.202852 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:06.202966 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 08:31:06.205892 22651 catalog_manager.cc:5495] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2602b66c20a4c90816bfe3ce31dde33 (127.15.134.194). New cstate: current_term: 1 leader_uuid: "b2602b66c20a4c90816bfe3ce31dde33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf9069423b204f8087751367c3f1cbc1" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 36669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 45771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2602b66c20a4c90816bfe3ce31dde33" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35129 } health_report { overall_health: HEALTHY } } }
I20260928 08:31:06.230695 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.225065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40910 (local address 127.15.134.194:35129)
0928 08:31:06.225376 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:06.225381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:06.225402 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:31:06.225588 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:06.225593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:06.225683 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:06.225834 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:31:06.226221 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.227156 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.228432 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.228938 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.229041 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:06.229434 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:31:06.230006 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:31:06.230120 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:06.230287 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 08:31:06.230361 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 08:31:06.234974 23263 raft_consensus.cc:1261] T fd7058748e3a4bd68544b715cafe2c82 P cf9069423b204f8087751367c3f1cbc1 [term 1 FOLLOWER]: Refusing update from remote peer b2602b66c20a4c90816bfe3ce31dde33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:31:06.235409 22974 raft_consensus.cc:1261] T fd7058748e3a4bd68544b715cafe2c82 P 5d00b51460aa4a2e93d8f6f353224a8a [term 1 FOLLOWER]: Refusing update from remote peer b2602b66c20a4c90816bfe3ce31dde33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:31:06.235773 23418 consensus_queue.cc:1035] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf9069423b204f8087751367c3f1cbc1" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 36669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:31:06.236239 23413 consensus_queue.cc:1035] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d00b51460aa4a2e93d8f6f353224a8a" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:31:06.239413 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.238373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.15.134.195:48295)
0928 08:31:06.238973 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:06.239090 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488}
I20260928 08:31:06.240129 23416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.238728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.15.134.193:55397)
0928 08:31:06.239757 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:06.239905 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":877}
I20260928 08:31:06.245813 23416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.245264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.15.134.193:53095)
0928 08:31:06.245527 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:06.245626 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260928 08:31:06.247829 23422 mvcc.cc:205] Tried to move back new op lower bound from 7334233154491539456 to 7334233154369572864. Current Snapshot: MvccSnapshot[applied={T|T < 7334233154491539456}]
I20260928 08:31:06.248662 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.248039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.15.134.194:47753)
0928 08:31:06.248339 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:06.248435 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174}
I20260928 08:31:06.249938 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.249010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.15.134.195:48817)
0928 08:31:06.249617 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:06.249731 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460}
I20260928 08:31:06.261806 22651 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:31:06.262097 22651 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:31:06.267916 22651 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f424a72d7ff342a191e308a76893e581: Sending DeleteTablet for 3 replicas of tablet fd7058748e3a4bd68544b715cafe2c82
I20260928 08:31:06.269137 23243 tablet_service.cc:1514] Processing DeleteTablet for tablet fd7058748e3a4bd68544b715cafe2c82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:31:06 UTC) from {username='oryx', principal='oryx/127.15.134.253@KRBTEST.COM'} at 127.0.0.1:60360
I20260928 08:31:06.269157 22954 tablet_service.cc:1514] Processing DeleteTablet for tablet fd7058748e3a4bd68544b715cafe2c82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:31:06 UTC) from {username='oryx', principal='oryx/127.15.134.253@KRBTEST.COM'} at 127.0.0.1:41626
I20260928 08:31:06.269972 23431 tablet_replica.cc:330] T fd7058748e3a4bd68544b715cafe2c82 P cf9069423b204f8087751367c3f1cbc1: stopping tablet replica
I20260928 08:31:06.270241 23431 raft_consensus.cc:2229] T fd7058748e3a4bd68544b715cafe2c82 P cf9069423b204f8087751367c3f1cbc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:31:06.270644 23431 raft_consensus.cc:2258] T fd7058748e3a4bd68544b715cafe2c82 P cf9069423b204f8087751367c3f1cbc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:31:06.271250 23430 tablet_replica.cc:330] T fd7058748e3a4bd68544b715cafe2c82 P 5d00b51460aa4a2e93d8f6f353224a8a: stopping tablet replica
I20260928 08:31:06.271471 23430 raft_consensus.cc:2229] T fd7058748e3a4bd68544b715cafe2c82 P 5d00b51460aa4a2e93d8f6f353224a8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:31:06.271812 23430 raft_consensus.cc:2258] T fd7058748e3a4bd68544b715cafe2c82 P 5d00b51460aa4a2e93d8f6f353224a8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:31:06.271853 23102 tablet_service.cc:1514] Processing DeleteTablet for tablet fd7058748e3a4bd68544b715cafe2c82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:31:06 UTC) from {username='oryx', principal='oryx/127.15.134.253@KRBTEST.COM'} at 127.0.0.1:40900
I20260928 08:31:06.273118 23430 ts_tablet_manager.cc:1905] T fd7058748e3a4bd68544b715cafe2c82 P 5d00b51460aa4a2e93d8f6f353224a8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:31:06.273543 23431 ts_tablet_manager.cc:1905] T fd7058748e3a4bd68544b715cafe2c82 P cf9069423b204f8087751367c3f1cbc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:31:06.273627 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 22827
I20260928 08:31:06.280064 23430 ts_tablet_manager.cc:1918] T fd7058748e3a4bd68544b715cafe2c82 P 5d00b51460aa4a2e93d8f6f353224a8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:31:06.280064 23431 ts_tablet_manager.cc:1918] T fd7058748e3a4bd68544b715cafe2c82 P cf9069423b204f8087751367c3f1cbc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:31:06.280224 23430 log.cc:1196] T fd7058748e3a4bd68544b715cafe2c82 P 5d00b51460aa4a2e93d8f6f353224a8a: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-0/wal/wals/fd7058748e3a4bd68544b715cafe2c82
I20260928 08:31:06.280225 23431 log.cc:1196] T fd7058748e3a4bd68544b715cafe2c82 P cf9069423b204f8087751367c3f1cbc1: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/fd7058748e3a4bd68544b715cafe2c82
I20260928 08:31:06.280745 23430 ts_tablet_manager.cc:1939] T fd7058748e3a4bd68544b715cafe2c82 P 5d00b51460aa4a2e93d8f6f353224a8a: Deleting consensus metadata
I20260928 08:31:06.281495 23431 ts_tablet_manager.cc:1939] T fd7058748e3a4bd68544b715cafe2c82 P cf9069423b204f8087751367c3f1cbc1: Deleting consensus metadata
I20260928 08:31:06.282034 23432 tablet_replica.cc:330] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33: stopping tablet replica
I20260928 08:31:06.282011 22637 catalog_manager.cc:4841] TS 5d00b51460aa4a2e93d8f6f353224a8a (127.15.134.193:45771): tablet fd7058748e3a4bd68544b715cafe2c82 (table test-table [id=47cb746187b949efb45008b204c6006c]) successfully deleted
I20260928 08:31:06.282234 23432 raft_consensus.cc:2229] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:31:06.282590 22636 catalog_manager.cc:4841] TS cf9069423b204f8087751367c3f1cbc1 (127.15.134.195:36669): tablet fd7058748e3a4bd68544b715cafe2c82 (table test-table [id=47cb746187b949efb45008b204c6006c]) successfully deleted
I20260928 08:31:06.283028 23432 raft_consensus.cc:2258] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:31:06.284298 23432 ts_tablet_manager.cc:1905] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:31:06.291697 23432 ts_tablet_manager.cc:1918] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:31:06.291939 23432 log.cc:1196] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/fd7058748e3a4bd68544b715cafe2c82
I20260928 08:31:06.292544 23432 ts_tablet_manager.cc:1939] T fd7058748e3a4bd68544b715cafe2c82 P b2602b66c20a4c90816bfe3ce31dde33: Deleting consensus metadata
I20260928 08:31:06.293979 22637 catalog_manager.cc:4841] TS b2602b66c20a4c90816bfe3ce31dde33 (127.15.134.194:35129): tablet fd7058748e3a4bd68544b715cafe2c82 (table test-table [id=47cb746187b949efb45008b204c6006c]) successfully deleted
I20260928 08:31:06.298976 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.275613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41636 (local address 127.15.134.193:45771)
0928 08:31:06.275805 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:06.275811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:06.281200 (+ 5389us) server_negotiation.cc:407] Connection header received
0928 08:31:06.281376 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:06.281382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:06.281463 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:06.281563 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:06.284223 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.285322 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.287145 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.287779 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.287867 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:06.291341 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:06.291360 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:06.291383 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:06.291432 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:06.295371 (+ 3939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:06.296356 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:06.296368 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:06.296373 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:06.296543 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:06.297233 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:06.297240 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:06.297243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:06.297733 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:06.298035 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:06.298361 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 08:31:06.298595 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 08:31:06.299746 23014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:06.444600 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 22827
I20260928 08:31:06.468921 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 23030
I20260928 08:31:06.473506 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.473029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.0.0.1:52100)
0928 08:31:06.473212 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:06.473326 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 08:31:06.474632 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.474179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:44257 (local address 127.0.0.1:52114)
0928 08:31:06.474341 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:06.474455 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:44257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 08:31:06.486080 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.469756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40924 (local address 127.15.134.194:35129)
0928 08:31:06.469931 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:06.469938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:06.469977 (+ 39us) server_negotiation.cc:407] Connection header received
0928 08:31:06.470327 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:06.470337 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:06.470439 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:06.470561 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:06.472085 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.473205 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.474933 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.475522 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.475662 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:06.478416 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:06.478429 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:06.478448 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:06.478490 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:06.482973 (+ 4483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:06.483800 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:06.483810 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:06.483813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:06.484001 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:06.484420 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:06.484425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:06.484429 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:06.484942 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:06.485285 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:06.485456 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 08:31:06.485677 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:31:06.486860 23162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:06.611040 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 23030
I20260928 08:31:06.631417 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 23180
I20260928 08:31:06.646283 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:06.632208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60370 (local address 127.15.134.195:36669)
0928 08:31:06.632386 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:06.632390 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:06.632424 (+ 34us) server_negotiation.cc:407] Connection header received
0928 08:31:06.632592 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:06.632596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:06.632727 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:06.632835 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:06.634325 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.635129 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.636445 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:06.637081 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:06.637202 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:06.639918 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:06.639935 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:06.639953 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:06.639990 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:06.643149 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:06.644035 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:06.644045 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:06.644051 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:06.644211 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:06.644841 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:06.644847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:06.644851 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:06.645283 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:06.645515 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:06.645855 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 08:31:06.646028 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 08:31:06.646989 23303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:06.743422 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 23180
I20260928 08:31:06.762943 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 22513
I20260928 08:31:06.764192 22681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:06.888495 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 22513
W20260928 08:31:06.910580 22825 connection.cc:535] server connection from 127.0.0.1:60748 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 08:31:06.910940 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 22614
I20260928 08:31:06.911969 22870 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:07.028600 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 22614
2026-09-28T08:31:07Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10047 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:31:07 dist-test-slave-qp2n krb5kdc[23441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:31:07 dist-test-slave-qp2n krb5kdc[23441](info): set up 2 sockets
Sep 28 08:31:07 dist-test-slave-qp2n krb5kdc[23441](info): commencing operation
krb5kdc: starting...
W20260928 08:31:08.135458 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:31:08 dist-test-slave-qp2n krb5kdc[23441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:31:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:31:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.254.keytab.
Entry for principal oryx/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.254.keytab.
I20260928 08:31:08.365370 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:38601
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.254.keytab
--principal=oryx/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:38601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:31:08.712172 23454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:08.712472 23454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:08.712687 23454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:08.720141 23454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:31:08.720247 23454 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 08:31:08.720329 23454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:08.720445 23454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:08.720530 23454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:31:08.720646 23454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:31:08.732215 23454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:38601
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.254.keytab
--principal=oryx/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:38601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:08.733733 23454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:08.735194 23454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:08.748045 23460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:08.748107 23462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:08.748270 23459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:08.749104 23454 server_base.cc:1034] running on GCE node
I20260928 08:31:08.749955 23454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:08.751488 23454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:08.752759 23454 hybrid_clock.cc:648] HybridClock initialized: now 1790584268752686 us; error 49 us; skew 500 ppm
I20260928 08:31:08.753111 23454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:31:08 dist-test-slave-qp2n krb5kdc[23441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584268, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:31:08.756443 23454 init.cc:377] Logged in from keytab as oryx/127.15.134.254@KRBTEST.COM (short username oryx)
I20260928 08:31:08.758087 23454 webserver.cc:460] Webserver started at http://127.15.134.254:45991/ using document root <none> and password file <none>
I20260928 08:31:08.758965 23454 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:08.759102 23454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:08.759442 23454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:31:08.762498 23454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "272fa4a356d6455fa214dc28eb720db1"
format_stamp: "Formatted at 2026-09-28 08:31:08 on dist-test-slave-qp2n"
server_key: "8eced2ee6aeb6c4d72d41d9bca825d25"
server_key_iv: "e56939cb9b049b7bb342a4a2fdf58a32"
server_key_version: "encryptionkey@0"
I20260928 08:31:08.763352 23454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "272fa4a356d6455fa214dc28eb720db1"
format_stamp: "Formatted at 2026-09-28 08:31:08 on dist-test-slave-qp2n"
server_key: "8eced2ee6aeb6c4d72d41d9bca825d25"
server_key_iv: "e56939cb9b049b7bb342a4a2fdf58a32"
server_key_version: "encryptionkey@0"
I20260928 08:31:08.768517 23454 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 08:31:08.771950 23469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:08.773303 23454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:31:08.773483 23454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "272fa4a356d6455fa214dc28eb720db1"
format_stamp: "Formatted at 2026-09-28 08:31:08 on dist-test-slave-qp2n"
server_key: "8eced2ee6aeb6c4d72d41d9bca825d25"
server_key_iv: "e56939cb9b049b7bb342a4a2fdf58a32"
server_key_version: "encryptionkey@0"
I20260928 08:31:08.773708 23454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:08.808975 23454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:08.813207 23454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:08.813525 23454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:08.836265 23454 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:38601
I20260928 08:31:08.836288 23530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:38601 every 8 connection(s)
I20260928 08:31:08.837767 23454 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:31:08.841574 23531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:08.848179 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 23454
I20260928 08:31:08.848404 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:31:08.849260 15899 external_mini_cluster.cc:1428] Setting key a4e4f8c440c1466758fe37b1e0a8770f
I20260928 08:31:08.852556 23531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1: Bootstrap starting.
I20260928 08:31:08.856105 23531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:08.857503 23531 log.cc:824] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1: Log is configured to *not* fsync() on all Append() calls
I20260928 08:31:08.860646 23531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1: No bootstrap required, opened a new log
Sep 28 08:31:08 dist-test-slave-qp2n krb5kdc[23441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM
I20260928 08:31:08.870359 23531 raft_consensus.cc:348] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "272fa4a356d6455fa214dc28eb720db1" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38601 } }
I20260928 08:31:08.870816 23531 raft_consensus.cc:374] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:08.870925 23531 raft_consensus.cc:729] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272fa4a356d6455fa214dc28eb720db1, State: Initialized, Role: FOLLOWER
I20260928 08:31:08.871605 23531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "272fa4a356d6455fa214dc28eb720db1" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38601 } }
I20260928 08:31:08.871870 23531 raft_consensus.cc:388] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:31:08.872009 23531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:31:08.872201 23531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:08.874593 23534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:08.851324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39882 (local address 127.15.134.254:38601)
0928 08:31:08.852129 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:08.852141 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:08.852182 (+ 41us) server_negotiation.cc:407] Connection header received
0928 08:31:08.853994 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:08.854029 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:08.854403 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:08.854728 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:08.856709 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:08.858734 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:08.860207 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:08.861286 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:08.861487 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:08.865598 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:08.865611 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:08.865634 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:08.865695 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:08.869939 (+ 4244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:08.870680 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:08.870688 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:08.870696 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:08.870927 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:08.871395 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:08.871401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:08.871404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:08.872134 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:08.872474 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:08.872991 (+ 517us) server_negotiation.cc:299] Negotiation successful
0928 08:31:08.873460 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":394,"threads_started":1}
I20260928 08:31:08.874895 23531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "272fa4a356d6455fa214dc28eb720db1" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38601 } }
I20260928 08:31:08.875415 23531 leader_election.cc:302] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 272fa4a356d6455fa214dc28eb720db1; no voters:
I20260928 08:31:08.876106 23531 leader_election.cc:288] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:31:08.876255 23536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:31:08.877655 23536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1 [term 1 LEADER]: Becoming Leader. State: Replica: 272fa4a356d6455fa214dc28eb720db1, State: Running, Role: LEADER
I20260928 08:31:08.878331 23536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "272fa4a356d6455fa214dc28eb720db1" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38601 } }
I20260928 08:31:08.878788 23531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:31:08.881721 23537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "272fa4a356d6455fa214dc28eb720db1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "272fa4a356d6455fa214dc28eb720db1" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38601 } } }
I20260928 08:31:08.881824 23538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 272fa4a356d6455fa214dc28eb720db1. Latest consensus state: current_term: 1 leader_uuid: "272fa4a356d6455fa214dc28eb720db1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "272fa4a356d6455fa214dc28eb720db1" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38601 } } }
I20260928 08:31:08.882120 23537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1 [sys.catalog]: This master's current role is: LEADER
I20260928 08:31:08.884768 23538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1 [sys.catalog]: This master's current role is: LEADER
I20260928 08:31:08.885413 23542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:31:08.891115 23542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:31:08.900812 23542 catalog_manager.cc:1347] Generated new cluster ID: 67bcd06d829c464d8443f39edf6d73ed
I20260928 08:31:08.900955 23542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:31:08.915954 23542 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:31:08.917901 23542 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:31:08.938019 23542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1: Generated new TSK 0
I20260928 08:31:08.939200 23542 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.15.134.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.134.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.134.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.193.keytab.
Entry for principal oryx/127.15.134.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.193.keytab.
I20260928 08:31:09.000787 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:0
--local_ip_for_outbound_sockets=127.15.134.193
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:38601
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.193.keytab
--principal=oryx/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:31:09.340415 23557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:09.340799 23557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:09.340971 23557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:09.348491 23557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:09.348683 23557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:09.348829 23557 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:31:09.348968 23557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:31:09.359452 23557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.193.keytab
--principal=oryx/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.134.254:38601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:09.360991 23557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:09.362409 23557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:09.376495 23565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:09.376482 23562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:09.376642 23563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:09.378477 23557 server_base.cc:1034] running on GCE node
I20260928 08:31:09.379276 23557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:09.380301 23557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:09.381613 23557 hybrid_clock.cc:648] HybridClock initialized: now 1790584269381498 us; error 88 us; skew 500 ppm
I20260928 08:31:09.381963 23557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:31:09 dist-test-slave-qp2n krb5kdc[23441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584269, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:31:09.385628 23557 init.cc:377] Logged in from keytab as oryx/127.15.134.193@KRBTEST.COM (short username oryx)
I20260928 08:31:09.387286 23557 webserver.cc:460] Webserver started at http://127.15.134.193:34677/ using document root <none> and password file <none>
I20260928 08:31:09.388149 23557 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:09.388266 23557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:09.388665 23557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:31:09.391782 23557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-0/data/instance:
uuid: "b215502609054ce4a0ba9def4ab40533"
format_stamp: "Formatted at 2026-09-28 08:31:09 on dist-test-slave-qp2n"
server_key: "269b433cd403f3cffebd3c971e80ee70"
server_key_iv: "7e06cf0864e82eb2affbad50cd57c692"
server_key_version: "encryptionkey@0"
I20260928 08:31:09.392784 23557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance:
uuid: "b215502609054ce4a0ba9def4ab40533"
format_stamp: "Formatted at 2026-09-28 08:31:09 on dist-test-slave-qp2n"
server_key: "269b433cd403f3cffebd3c971e80ee70"
server_key_iv: "7e06cf0864e82eb2affbad50cd57c692"
server_key_version: "encryptionkey@0"
I20260928 08:31:09.398419 23557 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:31:09.402163 23572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:09.403566 23557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:31:09.403767 23557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "b215502609054ce4a0ba9def4ab40533"
format_stamp: "Formatted at 2026-09-28 08:31:09 on dist-test-slave-qp2n"
server_key: "269b433cd403f3cffebd3c971e80ee70"
server_key_iv: "7e06cf0864e82eb2affbad50cd57c692"
server_key_version: "encryptionkey@0"
I20260928 08:31:09.403995 23557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:09.418188 23557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:09.422297 23557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:09.422570 23557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:09.425146 23557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:31:09.425252 23557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:09.425431 23557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:31:09.425508 23557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:31:09 dist-test-slave-qp2n krb5kdc[23441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584269, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM
I20260928 08:31:09.450836 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:09.426032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:55679 (local address 127.15.134.254:38601)
0928 08:31:09.426530 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:09.426538 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:09.427713 (+ 1175us) server_negotiation.cc:407] Connection header received
0928 08:31:09.430046 (+ 2333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:09.430066 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:09.430171 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:09.430272 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:09.432793 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:09.434122 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:09.436789 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:09.437720 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:09.437891 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:09.443220 (+ 5329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:09.443226 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:09.443229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:09.443264 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:09.446756 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:09.447761 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:09.447767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:09.447771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:09.447924 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:09.448604 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:09.448635 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:09.448641 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:09.449052 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:09.449236 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:09.450211 (+ 975us) server_negotiation.cc:299] Negotiation successful
0928 08:31:09.450449 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":242,"threads_started":1}
I20260928 08:31:09.452030 23582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:09.426648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:38601 (local address 127.15.134.193:55679)
0928 08:31:09.427511 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:09.427566 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:09.429687 (+ 2121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:09.430848 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:09.430866 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:09.431697 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:09.432521 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:09.432591 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:09.434314 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:09.434323 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:09.436579 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:09.436602 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:09.438041 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:09.438051 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:09.438379 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:09.439287 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:09.439330 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:09.442934 (+ 3604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:09.446918 (+ 3984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:09.446928 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:09.446951 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:09.447576 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:09.448082 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:09.448091 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:09.448097 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:09.448439 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:09.449326 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:09.449336 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:09.449797 (+ 461us) client_negotiation.cc:769] Sending connection context
0928 08:31:09.450119 (+ 322us) client_negotiation.cc:240] Negotiation successful
0928 08:31:09.450737 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":374,"threads_started":1}
I20260928 08:31:09.475062 23655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:09.458305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:38601 (local address 127.15.134.193:48613)
0928 08:31:09.459046 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:09.459073 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:09.459254 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:09.459645 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:09.459654 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:09.460418 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:09.461103 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:09.461184 (+ 81us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:09.462336 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:09.462345 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:09.464430 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:09.464453 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:09.465594 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:09.465601 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:09.465845 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:09.466728 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:09.466757 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:09.468292 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:09.472041 (+ 3749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:09.472046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:09.472050 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:09.472705 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:09.473219 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:09.473222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:09.473224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:09.473346 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:09.474076 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:09.474079 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:09.474323 (+ 244us) client_negotiation.cc:769] Sending connection context
0928 08:31:09.474452 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 08:31:09.474719 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":350,"threads_started":1}
I20260928 08:31:09.475457 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:09.458348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:48613 (local address 127.15.134.254:38601)
0928 08:31:09.458597 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:09.458606 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:09.459181 (+ 575us) server_negotiation.cc:407] Connection header received
0928 08:31:09.459415 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:09.459419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:09.459486 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:09.459576 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:09.461366 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:09.462176 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:09.464670 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:09.465402 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:09.465543 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:09.468659 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:09.468668 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:09.468673 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:09.468725 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:09.471898 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:09.472848 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:09.472857 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:09.472863 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:09.473070 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:09.473457 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:09.473465 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:09.473471 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:09.473951 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:09.474193 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:09.474613 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 08:31:09.474933 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 08:31:09.482769 23557 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:44847
I20260928 08:31:09.482787 23692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:44847 every 8 connection(s)
I20260928 08:31:09.484017 23557 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:31:09.486109 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 23557
I20260928 08:31:09.486153 23693 heartbeater.cc:347] Connected to a master server at 127.15.134.254:38601
I20260928 08:31:09.486331 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance
I20260928 08:31:09.486440 23693 heartbeater.cc:464] Registering TS with master...
I20260928 08:31:09.487185 15899 external_mini_cluster.cc:1428] Setting key 0cb16916fe29d9e5d49716bd34aac45a
I20260928 08:31:09.487290 23693 heartbeater.cc:511] Master 127.15.134.254:38601 requested a full tablet report, sending...
I20260928 08:31:09.489660 23485 ts_manager.cc:194] Registered new tserver with Master: b215502609054ce4a0ba9def4ab40533 (127.15.134.193:44847)
I20260928 08:31:09.491115 23485 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.134.193@KRBTEST.COM'} at 127.15.134.193:55679
WARNING: no policy specified for oryx/127.15.134.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.134.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.134.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.194.keytab.
Entry for principal oryx/127.15.134.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.194.keytab.
I20260928 08:31:09.536016 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:0
--local_ip_for_outbound_sockets=127.15.134.194
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:38601
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.194.keytab
--principal=oryx/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:31:09.875260 23697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:09.875658 23697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:09.875830 23697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:09.883345 23697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:09.883494 23697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:09.883652 23697 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:31:09.883790 23697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:31:09.894090 23697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.194.keytab
--principal=oryx/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.134.254:38601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:09.895452 23697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:09.896934 23697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:09.909958 23703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:09.909993 23705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:09.910410 23702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:09.911073 23697 server_base.cc:1034] running on GCE node
I20260928 08:31:09.911674 23697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:09.912577 23697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:09.913861 23697 hybrid_clock.cc:648] HybridClock initialized: now 1790584269913804 us; error 31 us; skew 500 ppm
I20260928 08:31:09.914175 23697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:31:09 dist-test-slave-qp2n krb5kdc[23441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584269, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:31:09.917375 23697 init.cc:377] Logged in from keytab as oryx/127.15.134.194@KRBTEST.COM (short username oryx)
I20260928 08:31:09.918751 23697 webserver.cc:460] Webserver started at http://127.15.134.194:45887/ using document root <none> and password file <none>
I20260928 08:31:09.919404 23697 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:09.919486 23697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:09.919734 23697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:31:09.922515 23697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-1/data/instance:
uuid: "4cf012777c4f49a3a8d81f43278ac323"
format_stamp: "Formatted at 2026-09-28 08:31:09 on dist-test-slave-qp2n"
server_key: "ba1bdabf74cabaaaca842a60d65749a0"
server_key_iv: "c6d23b711723f3622fb82c75cfa97acd"
server_key_version: "encryptionkey@0"
I20260928 08:31:09.923241 23697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance:
uuid: "4cf012777c4f49a3a8d81f43278ac323"
format_stamp: "Formatted at 2026-09-28 08:31:09 on dist-test-slave-qp2n"
server_key: "ba1bdabf74cabaaaca842a60d65749a0"
server_key_iv: "c6d23b711723f3622fb82c75cfa97acd"
server_key_version: "encryptionkey@0"
I20260928 08:31:09.928257 23697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:31:09.931785 23712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:09.933328 23697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:31:09.933537 23697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "4cf012777c4f49a3a8d81f43278ac323"
format_stamp: "Formatted at 2026-09-28 08:31:09 on dist-test-slave-qp2n"
server_key: "ba1bdabf74cabaaaca842a60d65749a0"
server_key_iv: "c6d23b711723f3622fb82c75cfa97acd"
server_key_version: "encryptionkey@0"
I20260928 08:31:09.933795 23697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:09.955835 23697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:09.960220 23697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:09.960493 23697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:09.963330 23697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:31:09.963438 23697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:09.963589 23697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:31:09.963670 23697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:31:09 dist-test-slave-qp2n krb5kdc[23441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584269, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM
I20260928 08:31:09.991825 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:09.963774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:50991 (local address 127.15.134.254:38601)
0928 08:31:09.964012 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:09.964022 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:09.965686 (+ 1664us) server_negotiation.cc:407] Connection header received
0928 08:31:09.968889 (+ 3203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:09.968897 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:09.968985 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:09.969093 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:09.971759 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:09.972507 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:09.976025 (+ 3518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:09.976637 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:09.976815 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:09.983440 (+ 6625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:09.983446 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:09.983450 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:09.983488 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:09.987288 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:09.988666 (+ 1378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:09.988676 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:09.988682 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:09.988888 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:09.989402 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:09.989410 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:09.989416 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:09.989930 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:09.990123 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:09.990927 (+ 804us) server_negotiation.cc:299] Negotiation successful
0928 08:31:09.991254 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:31:09.992471 23721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:09.964480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:38601 (local address 127.15.134.194:50991)
0928 08:31:09.965434 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:09.965494 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:09.968466 (+ 2972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:09.969462 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:09.969479 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:09.970423 (+ 944us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:09.971496 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:09.971534 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:09.972700 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:09.972709 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:09.975834 (+ 3125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:09.975860 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:09.976938 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:09.976950 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:09.977366 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:09.978352 (+ 986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:09.978402 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:09.983116 (+ 4714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:09.987526 (+ 4410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:09.987573 (+ 47us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:09.987605 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:09.988447 (+ 842us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:09.989038 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:09.989042 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:09.989044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:09.989267 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:09.990084 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:09.990091 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:09.990484 (+ 393us) client_negotiation.cc:769] Sending connection context
0928 08:31:09.990740 (+ 256us) client_negotiation.cc:240] Negotiation successful
0928 08:31:09.991169 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":374,"threads_started":1}
I20260928 08:31:10.020952 23697 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:34495
I20260928 08:31:10.020917 23831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:34495 every 8 connection(s)
I20260928 08:31:10.022225 23788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:10.000879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:38601 (local address 127.15.134.194:53371)
0928 08:31:10.005367 (+ 4488us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:10.005391 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:10.005633 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:10.006126 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:10.006134 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:10.007214 (+ 1080us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:10.007977 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:10.007994 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.009787 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.009798 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:10.010940 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:10.010953 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.012140 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.012147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:10.012275 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:10.013006 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:10.013024 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:10.014397 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:10.019152 (+ 4755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:10.019156 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:10.019159 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:10.019718 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:10.020224 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:10.020230 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:10.020236 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:10.020457 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:10.021311 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:10.021314 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:10.021477 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 08:31:10.021644 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 08:31:10.021909 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4316,"thread_start_us":4140,"threads_started":1}
I20260928 08:31:10.022368 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:10.000836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:53371 (local address 127.15.134.254:38601)
0928 08:31:10.001084 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:10.001093 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:10.005538 (+ 4445us) server_negotiation.cc:407] Connection header received
0928 08:31:10.005797 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:10.005805 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:10.005918 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:10.006030 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:10.008219 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.009574 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.011100 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.011904 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.012045 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:10.014666 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:10.014674 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:10.014682 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:10.014730 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:10.019022 (+ 4292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:10.019862 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:10.019868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:10.019872 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:10.020061 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:10.020696 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:10.020704 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:10.020710 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:10.021187 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:10.021342 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:10.021771 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 08:31:10.021999 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 08:31:10.022627 23697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:31:10.024685 23832 heartbeater.cc:347] Connected to a master server at 127.15.134.254:38601
I20260928 08:31:10.025992 23832 heartbeater.cc:464] Registering TS with master...
I20260928 08:31:10.026749 23832 heartbeater.cc:511] Master 127.15.134.254:38601 requested a full tablet report, sending...
I20260928 08:31:10.028208 23485 ts_manager.cc:194] Registered new tserver with Master: 4cf012777c4f49a3a8d81f43278ac323 (127.15.134.194:34495)
I20260928 08:31:10.028687 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 23697
I20260928 08:31:10.028856 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance
I20260928 08:31:10.029218 23485 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.134.194@KRBTEST.COM'} at 127.15.134.194:50991
I20260928 08:31:10.029305 15899 external_mini_cluster.cc:1428] Setting key 9031f0955ee09080e0ae004afc7d638a
WARNING: no policy specified for oryx/127.15.134.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.134.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.134.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.195.keytab.
Entry for principal oryx/127.15.134.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.195.keytab.
I20260928 08:31:10.080533 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:0
--local_ip_for_outbound_sockets=127.15.134.195
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:38601
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.195.keytab
--principal=oryx/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:36541
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:31:10.413160 23836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:10.413441 23836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:10.413609 23836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:10.423766 23836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:10.423908 23836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:10.424070 23836 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:31:10.424218 23836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:31:10.434598 23836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:36541
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.195.keytab
--principal=oryx/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.15.134.254:38601
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:10.435971 23836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:10.437717 23836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:10.451263 23841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:10.451294 23842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:10.451262 23844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:10.452968 23836 server_base.cc:1034] running on GCE node
I20260928 08:31:10.453771 23836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:10.454787 23836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:10.456077 23836 hybrid_clock.cc:648] HybridClock initialized: now 1790584270455958 us; error 59 us; skew 500 ppm
I20260928 08:31:10.456411 23836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:31:10 dist-test-slave-qp2n krb5kdc[23441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584270, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:31:10.459808 23836 init.cc:377] Logged in from keytab as oryx/127.15.134.195@KRBTEST.COM (short username oryx)
I20260928 08:31:10.461504 23836 webserver.cc:460] Webserver started at http://127.15.134.195:42785/ using document root <none> and password file <none>
I20260928 08:31:10.462261 23836 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:10.462396 23836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:10.462810 23836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:31:10.465842 23836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-2/data/instance:
uuid: "3fdb1684336f4d56b74b80a45e1dca14"
format_stamp: "Formatted at 2026-09-28 08:31:10 on dist-test-slave-qp2n"
server_key: "7197ef7a439eaa9d620a39371241bfef"
server_key_iv: "0b37b38c745a6dff36d34e4cf07f877b"
server_key_version: "encryptionkey@0"
I20260928 08:31:10.466694 23836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance:
uuid: "3fdb1684336f4d56b74b80a45e1dca14"
format_stamp: "Formatted at 2026-09-28 08:31:10 on dist-test-slave-qp2n"
server_key: "7197ef7a439eaa9d620a39371241bfef"
server_key_iv: "0b37b38c745a6dff36d34e4cf07f877b"
server_key_version: "encryptionkey@0"
I20260928 08:31:10.472224 23836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 08:31:10.476022 23851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:10.477836 23836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:31:10.478040 23836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "3fdb1684336f4d56b74b80a45e1dca14"
format_stamp: "Formatted at 2026-09-28 08:31:10 on dist-test-slave-qp2n"
server_key: "7197ef7a439eaa9d620a39371241bfef"
server_key_iv: "0b37b38c745a6dff36d34e4cf07f877b"
server_key_version: "encryptionkey@0"
I20260928 08:31:10.478341 23836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:10.494733 23693 heartbeater.cc:503] Master 127.15.134.254:38601 was elected leader, sending a full tablet report...
I20260928 08:31:10.507176 23836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:10.511507 23836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:10.511734 23836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:10.514349 23836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:31:10.514474 23836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:10.514606 23836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:31:10.514690 23836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:31:10 dist-test-slave-qp2n krb5kdc[23441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584270, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM
I20260928 08:31:10.541955 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:10.515234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:41285 (local address 127.15.134.254:38601)
0928 08:31:10.515423 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:10.515428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:10.517037 (+ 1609us) server_negotiation.cc:407] Connection header received
0928 08:31:10.519893 (+ 2856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:10.519903 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:10.520015 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:10.520142 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:10.522451 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.523646 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.526629 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.527336 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.527498 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:10.533119 (+ 5621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:10.533125 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:10.533129 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:10.533165 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:10.537163 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:10.538515 (+ 1352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:10.538525 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:10.538531 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:10.538732 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:10.539580 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:10.539584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:10.539586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:10.539885 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:10.540077 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:10.541254 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0928 08:31:10.541516 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 08:31:10.543715 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:10.515842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:38601 (local address 127.15.134.195:41285)
0928 08:31:10.516855 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:10.516904 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:10.519513 (+ 2609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:10.520397 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:10.520405 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:10.521168 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:10.522194 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:10.522239 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.523861 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.523866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:10.526387 (+ 2521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:10.526412 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.527854 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.527869 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:10.528235 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:10.529172 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:10.529197 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:10.532790 (+ 3593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:10.537371 (+ 4581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:10.537386 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:10.537410 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:10.538330 (+ 920us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:10.538922 (+ 592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:10.538927 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:10.538932 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:10.539323 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:10.540060 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:10.540069 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:10.540675 (+ 606us) client_negotiation.cc:769] Sending connection context
0928 08:31:10.541103 (+ 428us) client_negotiation.cc:240] Negotiation successful
0928 08:31:10.541770 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":437,"threads_started":1}
I20260928 08:31:10.565577 23836 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:34143
I20260928 08:31:10.565603 23970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:34143 every 8 connection(s)
I20260928 08:31:10.566852 23836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:31:10.568816 23948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:10.552253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:38601 (local address 127.15.134.195:50681)
0928 08:31:10.553232 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:10.553259 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:10.553429 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:10.553827 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:10.553832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:10.554433 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:10.554763 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:10.554773 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.555864 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.555870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:10.557338 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:10.557364 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.558498 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.558510 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:10.558810 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:10.559686 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:10.559722 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:10.561124 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:10.565141 (+ 4017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:10.565147 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:10.565152 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:10.565700 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:10.566173 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:10.566178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:10.566182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:10.566322 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:10.567216 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:10.567222 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:10.567518 (+ 296us) client_negotiation.cc:769] Sending connection context
0928 08:31:10.567727 (+ 209us) client_negotiation.cc:240] Negotiation successful
0928 08:31:10.568175 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"thread_start_us":610,"threads_started":1}
I20260928 08:31:10.569136 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:10.552323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:50681 (local address 127.15.134.254:38601)
0928 08:31:10.552563 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:10.552573 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:10.553353 (+ 780us) server_negotiation.cc:407] Connection header received
0928 08:31:10.553594 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:10.553599 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:10.553667 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:10.553779 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:10.554924 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.555697 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.557550 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.558224 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.558387 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:10.561413 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:10.561419 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:10.561422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:10.561458 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:10.564984 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:10.565884 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:10.565889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:10.565891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:10.566037 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:10.566452 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:10.566458 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:10.566461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:10.566866 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:10.567313 (+ 447us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:10.567907 (+ 594us) server_negotiation.cc:299] Negotiation successful
0928 08:31:10.568196 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 08:31:10.570062 23971 heartbeater.cc:347] Connected to a master server at 127.15.134.254:38601
I20260928 08:31:10.570538 23971 heartbeater.cc:464] Registering TS with master...
I20260928 08:31:10.571537 23971 heartbeater.cc:511] Master 127.15.134.254:38601 requested a full tablet report, sending...
I20260928 08:31:10.573199 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 23836
I20260928 08:31:10.573396 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance
I20260928 08:31:10.573601 23485 ts_manager.cc:194] Registered new tserver with Master: 3fdb1684336f4d56b74b80a45e1dca14 (127.15.134.195:34143)
I20260928 08:31:10.573987 15899 external_mini_cluster.cc:1428] Setting key 5bbdc55069b480b74820131d386b95c5
I20260928 08:31:10.574523 23485 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.15.134.195@KRBTEST.COM'} at 127.15.134.195:41285
I20260928 08:31:10.577076 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 08:31:10 dist-test-slave-qp2n krb5kdc[23441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584270, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 08:31:10 dist-test-slave-qp2n krb5kdc[23441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:10 dist-test-slave-qp2n krb5kdc[23441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 08:31:10.630581 23581 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:10.619725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39884 (local address 127.15.134.254:38601)
0928 08:31:10.619970 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:10.619977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:10.620241 (+ 264us) server_negotiation.cc:407] Connection header received
0928 08:31:10.620457 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:10.620462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:10.620540 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:10.620686 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:10.622108 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.622841 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.624082 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.624482 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.624622 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:10.630255 (+ 5633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39884: BlockingRecv error: recv got EOF from 127.0.0.1:39884 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 28 08:31:10 dist-test-slave-qp2n krb5kdc[23441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584270, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM
I20260928 08:31:10.648376 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:10.633867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39898 (local address 127.15.134.254:38601)
0928 08:31:10.634101 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:10.634108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:10.634383 (+ 275us) server_negotiation.cc:407] Connection header received
0928 08:31:10.634600 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:10.634604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:10.634665 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:10.634771 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:10.635932 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.636782 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.638206 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.638747 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.638890 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:10.642745 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:10.642751 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:10.642755 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:10.642792 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:10.645694 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:10.646564 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:10.646568 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:10.646570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:10.646737 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:10.647174 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:10.647177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:10.647179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:10.647448 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:10.647731 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:10.647962 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 08:31:10.648139 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 08:31:10.652788 23485 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:31:10.659544 23485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:31:10.683102 23990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:10.671841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37728 (local address 127.15.134.194:34495)
0928 08:31:10.673450 (+ 1609us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:10.673457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:10.673483 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:31:10.673575 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:10.673583 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:10.673853 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:10.674976 (+ 1123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:10.675212 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.677184 (+ 1972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.680321 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.681786 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.682016 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:10.682275 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:10.682619 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 08:31:10.682797 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1459,"thread_start_us":1337,"threads_started":1}
I20260928 08:31:10.683157 23989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:10.671572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:34495 (local address 127.0.0.1:37728)
0928 08:31:10.672496 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:10.672557 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:10.672856 (+ 299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:10.674072 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:10.674083 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:10.674125 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:10.674711 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:10.674730 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.677378 (+ 2648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.677387 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:10.680138 (+ 2751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:10.680153 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.682066 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.682073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:10.682293 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:10.682353 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 08:31:10.682495 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 08:31:10.682749 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":2,"thread_start_us":409,"threads_started":1}
I20260928 08:31:10.685101 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:10.672500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55036 (local address 127.15.134.195:34143)
0928 08:31:10.674008 (+ 1508us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:10.674019 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:10.674843 (+ 824us) server_negotiation.cc:407] Connection header received
0928 08:31:10.674954 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:10.674963 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:10.675225 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:10.676674 (+ 1449us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:10.678517 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.680161 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.682539 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.683960 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.684169 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:10.684336 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:10.684565 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 08:31:10.684758 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1302,"thread_start_us":1133,"threads_started":1}
I20260928 08:31:10.686959 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:10.674693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58582 (local address 127.15.134.193:44847)
0928 08:31:10.678855 (+ 4162us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:10.678865 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:10.678894 (+ 29us) server_negotiation.cc:407] Connection header received
0928 08:31:10.678996 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:10.679005 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:10.679309 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:10.680274 (+ 965us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:10.680475 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.681461 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.684435 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.685951 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.686124 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:10.686278 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:10.686489 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 08:31:10.686669 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4026,"thread_start_us":3900,"threads_started":1}
I20260928 08:31:10.687000 23991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:10.671852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:34143 (local address 127.0.0.1:55036)
0928 08:31:10.674558 (+ 2706us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:10.674578 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:10.674713 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:10.677099 (+ 2386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:10.677106 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:10.677140 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:10.678279 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:10.678293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.680321 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.680326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:10.682385 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:10.682397 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.684182 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.684189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:10.684346 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:10.684392 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 08:31:10.684479 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 08:31:10.686579 (+ 2100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2568,"mutex_wait_us":552,"thread_start_us":2402,"threads_started":1}
I20260928 08:31:10.687021 23993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:10.674028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:44847 (local address 127.0.0.1:58582)
0928 08:31:10.678088 (+ 4060us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:10.678115 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:10.678272 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:10.679524 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:10.679528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:10.679554 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:10.680035 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:10.680049 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.681599 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.681605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:10.684244 (+ 2639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:10.684261 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.686178 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.686183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:10.686295 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:10.686318 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 08:31:10.686434 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 08:31:10.686560 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3886,"mutex_wait_us":3,"thread_start_us":3670,"threads_started":1}
I20260928 08:31:10.687538 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 77238a3e549749ad8dcf2d18bfc9d64d (DEFAULT_TABLE table=test-table [id=8ba702fd506941b995d77900fe67daa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:31:10.689268 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77238a3e549749ad8dcf2d18bfc9d64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:10.690706 23901 tablet_service.cc:1467] Processing CreateTablet for tablet 77238a3e549749ad8dcf2d18bfc9d64d (DEFAULT_TABLE table=test-table [id=8ba702fd506941b995d77900fe67daa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:31:10.690830 23623 tablet_service.cc:1467] Processing CreateTablet for tablet 77238a3e549749ad8dcf2d18bfc9d64d (DEFAULT_TABLE table=test-table [id=8ba702fd506941b995d77900fe67daa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:31:10.691938 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77238a3e549749ad8dcf2d18bfc9d64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:10.692122 23623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77238a3e549749ad8dcf2d18bfc9d64d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:10.704779 23995 tablet_bootstrap.cc:492] T 77238a3e549749ad8dcf2d18bfc9d64d P 3fdb1684336f4d56b74b80a45e1dca14: Bootstrap starting.
I20260928 08:31:10.708040 23996 tablet_bootstrap.cc:492] T 77238a3e549749ad8dcf2d18bfc9d64d P 4cf012777c4f49a3a8d81f43278ac323: Bootstrap starting.
I20260928 08:31:10.708645 23995 tablet_bootstrap.cc:654] T 77238a3e549749ad8dcf2d18bfc9d64d P 3fdb1684336f4d56b74b80a45e1dca14: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:10.709102 23997 tablet_bootstrap.cc:492] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533: Bootstrap starting.
I20260928 08:31:10.710150 23995 log.cc:824] T 77238a3e549749ad8dcf2d18bfc9d64d P 3fdb1684336f4d56b74b80a45e1dca14: Log is configured to *not* fsync() on all Append() calls
I20260928 08:31:10.712525 23996 tablet_bootstrap.cc:654] T 77238a3e549749ad8dcf2d18bfc9d64d P 4cf012777c4f49a3a8d81f43278ac323: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:10.712760 23997 tablet_bootstrap.cc:654] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:10.713728 23995 tablet_bootstrap.cc:492] T 77238a3e549749ad8dcf2d18bfc9d64d P 3fdb1684336f4d56b74b80a45e1dca14: No bootstrap required, opened a new log
I20260928 08:31:10.713781 23996 log.cc:824] T 77238a3e549749ad8dcf2d18bfc9d64d P 4cf012777c4f49a3a8d81f43278ac323: Log is configured to *not* fsync() on all Append() calls
I20260928 08:31:10.714042 23995 ts_tablet_manager.cc:1397] T 77238a3e549749ad8dcf2d18bfc9d64d P 3fdb1684336f4d56b74b80a45e1dca14: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260928 08:31:10.714200 23997 log.cc:824] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533: Log is configured to *not* fsync() on all Append() calls
I20260928 08:31:10.717133 23997 tablet_bootstrap.cc:492] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533: No bootstrap required, opened a new log
I20260928 08:31:10.717293 23996 tablet_bootstrap.cc:492] T 77238a3e549749ad8dcf2d18bfc9d64d P 4cf012777c4f49a3a8d81f43278ac323: No bootstrap required, opened a new log
I20260928 08:31:10.717490 23997 ts_tablet_manager.cc:1397] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260928 08:31:10.717583 23996 ts_tablet_manager.cc:1397] T 77238a3e549749ad8dcf2d18bfc9d64d P 4cf012777c4f49a3a8d81f43278ac323: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 08:31:10.726095 23997 raft_consensus.cc:348] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } } peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } }
I20260928 08:31:10.726118 23995 raft_consensus.cc:348] T 77238a3e549749ad8dcf2d18bfc9d64d P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } } peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } }
I20260928 08:31:10.726553 23997 raft_consensus.cc:374] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:10.726568 23995 raft_consensus.cc:374] T 77238a3e549749ad8dcf2d18bfc9d64d P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:10.726665 23995 raft_consensus.cc:729] T 77238a3e549749ad8dcf2d18bfc9d64d P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fdb1684336f4d56b74b80a45e1dca14, State: Initialized, Role: FOLLOWER
I20260928 08:31:10.726765 23997 raft_consensus.cc:729] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b215502609054ce4a0ba9def4ab40533, State: Initialized, Role: FOLLOWER
I20260928 08:31:10.727439 23997 consensus_queue.cc:260] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } } peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } }
I20260928 08:31:10.727459 23995 consensus_queue.cc:260] T 77238a3e549749ad8dcf2d18bfc9d64d P 3fdb1684336f4d56b74b80a45e1dca14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } } peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } }
I20260928 08:31:10.729511 23997 ts_tablet_manager.cc:1428] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260928 08:31:10.729972 23996 raft_consensus.cc:348] T 77238a3e549749ad8dcf2d18bfc9d64d P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } } peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } }
I20260928 08:31:10.730571 23996 raft_consensus.cc:374] T 77238a3e549749ad8dcf2d18bfc9d64d P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:10.730629 23971 heartbeater.cc:503] Master 127.15.134.254:38601 was elected leader, sending a full tablet report...
I20260928 08:31:10.730693 23996 raft_consensus.cc:729] T 77238a3e549749ad8dcf2d18bfc9d64d P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cf012777c4f49a3a8d81f43278ac323, State: Initialized, Role: FOLLOWER
I20260928 08:31:10.731382 23996 consensus_queue.cc:260] T 77238a3e549749ad8dcf2d18bfc9d64d P 4cf012777c4f49a3a8d81f43278ac323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } } peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } }
I20260928 08:31:10.733062 23995 ts_tablet_manager.cc:1428] T 77238a3e549749ad8dcf2d18bfc9d64d P 3fdb1684336f4d56b74b80a45e1dca14: Time spent starting tablet: real 0.019s user 0.014s sys 0.004s
I20260928 08:31:10.734272 23832 heartbeater.cc:503] Master 127.15.134.254:38601 was elected leader, sending a full tablet report...
I20260928 08:31:10.735033 23996 ts_tablet_manager.cc:1428] T 77238a3e549749ad8dcf2d18bfc9d64d P 4cf012777c4f49a3a8d81f43278ac323: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
W20260928 08:31:10.736433 23694 tablet.cc:2369] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:31:10.775729 23833 tablet.cc:2369] T 77238a3e549749ad8dcf2d18bfc9d64d P 4cf012777c4f49a3a8d81f43278ac323: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:31:10.816143 24001 raft_consensus.cc:482] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:31:10.816519 24001 raft_consensus.cc:504] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } } peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } }
I20260928 08:31:10.818336 24001 leader_election.cc:288] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cf012777c4f49a3a8d81f43278ac323 (127.15.134.194:34495), 3fdb1684336f4d56b74b80a45e1dca14 (127.15.134.195:34143)
W20260928 08:31:10.820573 23972 tablet.cc:2369] T 77238a3e549749ad8dcf2d18bfc9d64d P 3fdb1684336f4d56b74b80a45e1dca14: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:31:10.827674 23990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:10.818862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:33503 (local address 127.15.134.194:34495)
0928 08:31:10.819064 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:10.819070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:10.820117 (+ 1047us) server_negotiation.cc:407] Connection header received
0928 08:31:10.820298 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:10.820302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:10.820360 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:10.820455 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:10.821134 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.822241 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.825417 (+ 3176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.826673 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.826865 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:10.826943 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:10.827277 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 08:31:10.827401 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 08:31:10.827772 24005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:10.819019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:34495 (local address 127.15.134.193:33503)
0928 08:31:10.820000 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:10.820027 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:10.820133 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:10.820475 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:10.820480 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:10.820504 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:10.820965 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:10.820978 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.822388 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.822393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:10.825223 (+ 2830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:10.825240 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.826908 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.826915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:10.827104 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:10.827134 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 08:31:10.827248 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 08:31:10.827384 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"mutex_wait_us":159,"thread_start_us":632,"threads_started":1}
I20260928 08:31:10.827872 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:10.818867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:54367 (local address 127.15.134.195:34143)
0928 08:31:10.819168 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:10.819175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:10.819859 (+ 684us) server_negotiation.cc:407] Connection header received
0928 08:31:10.820091 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:10.820098 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:10.820180 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:10.820347 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:10.821352 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.822372 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.825417 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.826751 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.826888 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:10.826950 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:10.827506 (+ 556us) server_negotiation.cc:299] Negotiation successful
0928 08:31:10.827633 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 08:31:10.828559 24004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:10.818875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:34143 (local address 127.15.134.193:54367)
0928 08:31:10.819698 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:10.819725 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:10.819920 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:10.820571 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:10.820576 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:10.820650 (+ 74us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:10.821188 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:10.821202 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.822537 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.822546 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:10.825223 (+ 2677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:10.825240 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.827151 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.827158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:10.827328 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:10.827360 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 08:31:10.828142 (+ 782us) client_negotiation.cc:240] Negotiation successful
0928 08:31:10.828271 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":204,"thread_start_us":388,"threads_started":1}
I20260928 08:31:10.829129 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77238a3e549749ad8dcf2d18bfc9d64d" candidate_uuid: "b215502609054ce4a0ba9def4ab40533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cf012777c4f49a3a8d81f43278ac323" is_pre_election: true
I20260928 08:31:10.829658 23786 raft_consensus.cc:2454] T 77238a3e549749ad8dcf2d18bfc9d64d P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b215502609054ce4a0ba9def4ab40533 in term 0.
I20260928 08:31:10.829591 23921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77238a3e549749ad8dcf2d18bfc9d64d" candidate_uuid: "b215502609054ce4a0ba9def4ab40533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fdb1684336f4d56b74b80a45e1dca14" is_pre_election: true
I20260928 08:31:10.830183 23921 raft_consensus.cc:2454] T 77238a3e549749ad8dcf2d18bfc9d64d P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b215502609054ce4a0ba9def4ab40533 in term 0.
I20260928 08:31:10.830353 23573 leader_election.cc:302] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cf012777c4f49a3a8d81f43278ac323, b215502609054ce4a0ba9def4ab40533; no voters:
I20260928 08:31:10.830871 24001 raft_consensus.cc:2781] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:31:10.831071 24001 raft_consensus.cc:482] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:31:10.831192 24001 raft_consensus.cc:3037] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:10.833576 24001 raft_consensus.cc:504] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } } peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } }
I20260928 08:31:10.834165 24001 leader_election.cc:288] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533 [CANDIDATE]: Term 1 election: Requested vote from peers 4cf012777c4f49a3a8d81f43278ac323 (127.15.134.194:34495), 3fdb1684336f4d56b74b80a45e1dca14 (127.15.134.195:34143)
I20260928 08:31:10.834743 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77238a3e549749ad8dcf2d18bfc9d64d" candidate_uuid: "b215502609054ce4a0ba9def4ab40533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cf012777c4f49a3a8d81f43278ac323"
I20260928 08:31:10.834743 23921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77238a3e549749ad8dcf2d18bfc9d64d" candidate_uuid: "b215502609054ce4a0ba9def4ab40533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fdb1684336f4d56b74b80a45e1dca14"
I20260928 08:31:10.834965 23921 raft_consensus.cc:3037] T 77238a3e549749ad8dcf2d18bfc9d64d P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:10.834966 23786 raft_consensus.cc:3037] T 77238a3e549749ad8dcf2d18bfc9d64d P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:10.838308 23921 raft_consensus.cc:2454] T 77238a3e549749ad8dcf2d18bfc9d64d P 3fdb1684336f4d56b74b80a45e1dca14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b215502609054ce4a0ba9def4ab40533 in term 1.
I20260928 08:31:10.838308 23786 raft_consensus.cc:2454] T 77238a3e549749ad8dcf2d18bfc9d64d P 4cf012777c4f49a3a8d81f43278ac323 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b215502609054ce4a0ba9def4ab40533 in term 1.
I20260928 08:31:10.838793 23576 leader_election.cc:302] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fdb1684336f4d56b74b80a45e1dca14, b215502609054ce4a0ba9def4ab40533; no voters:
I20260928 08:31:10.839184 24001 raft_consensus.cc:2781] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:31:10.839715 24001 raft_consensus.cc:686] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533 [term 1 LEADER]: Becoming Leader. State: Replica: b215502609054ce4a0ba9def4ab40533, State: Running, Role: LEADER
I20260928 08:31:10.840353 24001 consensus_queue.cc:237] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } } peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } }
I20260928 08:31:10.845912 23485 catalog_manager.cc:5495] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533 reported cstate change: term changed from 0 to 1, leader changed from <none> to b215502609054ce4a0ba9def4ab40533 (127.15.134.193). New cstate: current_term: 1 leader_uuid: "b215502609054ce4a0ba9def4ab40533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } health_report { overall_health: HEALTHY } } }
I20260928 08:31:10.880913 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:10.873349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58592 (local address 127.15.134.193:44847)
0928 08:31:10.873565 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:10.873572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:10.873674 (+ 102us) server_negotiation.cc:407] Connection header received
0928 08:31:10.873922 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:10.873929 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:10.874113 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:10.874283 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:31:10.874765 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.875933 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.878029 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:10.878747 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:10.878885 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:10.879489 (+ 604us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:31:10.880172 (+ 683us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:31:10.880281 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:10.880430 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 08:31:10.880528 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:31:10.890548 23921 raft_consensus.cc:1261] T 77238a3e549749ad8dcf2d18bfc9d64d P 3fdb1684336f4d56b74b80a45e1dca14 [term 1 FOLLOWER]: Refusing update from remote peer b215502609054ce4a0ba9def4ab40533: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:31:10.890538 23786 raft_consensus.cc:1261] T 77238a3e549749ad8dcf2d18bfc9d64d P 4cf012777c4f49a3a8d81f43278ac323 [term 1 FOLLOWER]: Refusing update from remote peer b215502609054ce4a0ba9def4ab40533: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:31:10.891745 24006 consensus_queue.cc:1035] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:31:10.892134 24001 consensus_queue.cc:1035] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:31:10.903759 24010 mvcc.cc:205] Tried to move back new op lower bound from 7334233173556391936 to 7334233173366927360. Current Snapshot: MvccSnapshot[applied={T|T < 7334233173556391936}]
I20260928 08:31:10.923614 23486 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:31:10.923874 23486 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:31:10.927924 23486 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1: Sending DeleteTablet for 3 replicas of tablet 77238a3e549749ad8dcf2d18bfc9d64d
I20260928 08:31:10.929162 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet 77238a3e549749ad8dcf2d18bfc9d64d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:31:10 UTC) from {username='oryx', principal='oryx/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:37728
I20260928 08:31:10.929288 23623 tablet_service.cc:1514] Processing DeleteTablet for tablet 77238a3e549749ad8dcf2d18bfc9d64d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:31:10 UTC) from {username='oryx', principal='oryx/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:58582
I20260928 08:31:10.929325 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 77238a3e549749ad8dcf2d18bfc9d64d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:31:10 UTC) from {username='oryx', principal='oryx/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:55036
I20260928 08:31:10.931241 23486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:31:10.932633 23486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:31:10.935768 24019 tablet_replica.cc:330] T 77238a3e549749ad8dcf2d18bfc9d64d P 4cf012777c4f49a3a8d81f43278ac323: stopping tablet replica
I20260928 08:31:10.936333 24019 raft_consensus.cc:2229] T 77238a3e549749ad8dcf2d18bfc9d64d P 4cf012777c4f49a3a8d81f43278ac323 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:31:10.937016 24019 raft_consensus.cc:2258] T 77238a3e549749ad8dcf2d18bfc9d64d P 4cf012777c4f49a3a8d81f43278ac323 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:31:10.942032 24019 ts_tablet_manager.cc:1905] T 77238a3e549749ad8dcf2d18bfc9d64d P 4cf012777c4f49a3a8d81f43278ac323: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:31:10.942507 23622 tablet_service.cc:1467] Processing CreateTablet for tablet 9ec1fd26e43b49969a4d05058080d9a8 (DEFAULT_TABLE table=test-table [id=4946a602ecff41e8abb3d39d959e9f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:31:10.942658 23900 tablet_service.cc:1467] Processing CreateTablet for tablet 9ec1fd26e43b49969a4d05058080d9a8 (DEFAULT_TABLE table=test-table [id=4946a602ecff41e8abb3d39d959e9f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:31:10.943295 23622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ec1fd26e43b49969a4d05058080d9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:10.943331 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ec1fd26e43b49969a4d05058080d9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:10.948104 23762 tablet_service.cc:1467] Processing CreateTablet for tablet 9ec1fd26e43b49969a4d05058080d9a8 (DEFAULT_TABLE table=test-table [id=4946a602ecff41e8abb3d39d959e9f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:31:10.948596 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ec1fd26e43b49969a4d05058080d9a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:10.952987 23995 tablet_bootstrap.cc:492] T 9ec1fd26e43b49969a4d05058080d9a8 P 3fdb1684336f4d56b74b80a45e1dca14: Bootstrap starting.
I20260928 08:31:10.955580 23995 tablet_bootstrap.cc:654] T 9ec1fd26e43b49969a4d05058080d9a8 P 3fdb1684336f4d56b74b80a45e1dca14: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:10.955914 24019 ts_tablet_manager.cc:1918] T 77238a3e549749ad8dcf2d18bfc9d64d P 4cf012777c4f49a3a8d81f43278ac323: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:31:10.956116 24019 log.cc:1196] T 77238a3e549749ad8dcf2d18bfc9d64d P 4cf012777c4f49a3a8d81f43278ac323: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/77238a3e549749ad8dcf2d18bfc9d64d
I20260928 08:31:10.956795 24019 ts_tablet_manager.cc:1939] T 77238a3e549749ad8dcf2d18bfc9d64d P 4cf012777c4f49a3a8d81f43278ac323: Deleting consensus metadata
I20260928 08:31:10.957718 23996 tablet_bootstrap.cc:492] T 9ec1fd26e43b49969a4d05058080d9a8 P 4cf012777c4f49a3a8d81f43278ac323: Bootstrap starting.
I20260928 08:31:10.958894 23470 catalog_manager.cc:4841] TS 4cf012777c4f49a3a8d81f43278ac323 (127.15.134.194:34495): tablet 77238a3e549749ad8dcf2d18bfc9d64d (table test-table [id=8ba702fd506941b995d77900fe67daa8]) successfully deleted
I20260928 08:31:10.960057 23996 tablet_bootstrap.cc:654] T 9ec1fd26e43b49969a4d05058080d9a8 P 4cf012777c4f49a3a8d81f43278ac323: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:10.960078 23997 tablet_bootstrap.cc:492] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533: Bootstrap starting.
I20260928 08:31:10.961664 23995 tablet_bootstrap.cc:492] T 9ec1fd26e43b49969a4d05058080d9a8 P 3fdb1684336f4d56b74b80a45e1dca14: No bootstrap required, opened a new log
I20260928 08:31:10.961864 23995 ts_tablet_manager.cc:1397] T 9ec1fd26e43b49969a4d05058080d9a8 P 3fdb1684336f4d56b74b80a45e1dca14: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 08:31:10.962306 23997 tablet_bootstrap.cc:654] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:10.963019 23995 raft_consensus.cc:348] T 9ec1fd26e43b49969a4d05058080d9a8 P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:10.963342 23995 raft_consensus.cc:374] T 9ec1fd26e43b49969a4d05058080d9a8 P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:10.963449 23995 raft_consensus.cc:729] T 9ec1fd26e43b49969a4d05058080d9a8 P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fdb1684336f4d56b74b80a45e1dca14, State: Initialized, Role: FOLLOWER
I20260928 08:31:10.963846 23995 consensus_queue.cc:260] T 9ec1fd26e43b49969a4d05058080d9a8 P 3fdb1684336f4d56b74b80a45e1dca14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:10.967758 23995 ts_tablet_manager.cc:1428] T 9ec1fd26e43b49969a4d05058080d9a8 P 3fdb1684336f4d56b74b80a45e1dca14: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260928 08:31:10.970396 24021 tablet_replica.cc:330] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533: stopping tablet replica
I20260928 08:31:10.970954 24021 raft_consensus.cc:2229] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:31:10.971230 23996 tablet_bootstrap.cc:492] T 9ec1fd26e43b49969a4d05058080d9a8 P 4cf012777c4f49a3a8d81f43278ac323: No bootstrap required, opened a new log
I20260928 08:31:10.971481 23996 ts_tablet_manager.cc:1397] T 9ec1fd26e43b49969a4d05058080d9a8 P 4cf012777c4f49a3a8d81f43278ac323: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260928 08:31:10.971623 24020 tablet_replica.cc:330] T 77238a3e549749ad8dcf2d18bfc9d64d P 3fdb1684336f4d56b74b80a45e1dca14: stopping tablet replica
I20260928 08:31:10.971843 24021 raft_consensus.cc:2258] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:31:10.972081 24020 raft_consensus.cc:2229] T 77238a3e549749ad8dcf2d18bfc9d64d P 3fdb1684336f4d56b74b80a45e1dca14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:31:10.972509 23997 tablet_bootstrap.cc:492] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533: No bootstrap required, opened a new log
I20260928 08:31:10.972604 24020 raft_consensus.cc:2258] T 77238a3e549749ad8dcf2d18bfc9d64d P 3fdb1684336f4d56b74b80a45e1dca14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:31:10.972523 23996 raft_consensus.cc:348] T 9ec1fd26e43b49969a4d05058080d9a8 P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:10.972714 23997 ts_tablet_manager.cc:1397] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.002s
I20260928 08:31:10.972792 23996 raft_consensus.cc:374] T 9ec1fd26e43b49969a4d05058080d9a8 P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:10.972893 23996 raft_consensus.cc:729] T 9ec1fd26e43b49969a4d05058080d9a8 P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cf012777c4f49a3a8d81f43278ac323, State: Initialized, Role: FOLLOWER
I20260928 08:31:10.973205 23996 consensus_queue.cc:260] T 9ec1fd26e43b49969a4d05058080d9a8 P 4cf012777c4f49a3a8d81f43278ac323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:10.973665 23997 raft_consensus.cc:348] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:10.973914 23997 raft_consensus.cc:374] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:10.974025 23997 raft_consensus.cc:729] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b215502609054ce4a0ba9def4ab40533, State: Initialized, Role: FOLLOWER
I20260928 08:31:10.974547 24021 ts_tablet_manager.cc:1905] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:31:10.975529 23997 consensus_queue.cc:260] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:10.976224 23997 ts_tablet_manager.cc:1428] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 08:31:10.976527 23996 ts_tablet_manager.cc:1428] T 9ec1fd26e43b49969a4d05058080d9a8 P 4cf012777c4f49a3a8d81f43278ac323: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260928 08:31:10.979271 24020 ts_tablet_manager.cc:1905] T 77238a3e549749ad8dcf2d18bfc9d64d P 3fdb1684336f4d56b74b80a45e1dca14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:31:10.986081 24020 ts_tablet_manager.cc:1918] T 77238a3e549749ad8dcf2d18bfc9d64d P 3fdb1684336f4d56b74b80a45e1dca14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:31:10.986303 24020 log.cc:1196] T 77238a3e549749ad8dcf2d18bfc9d64d P 3fdb1684336f4d56b74b80a45e1dca14: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/77238a3e549749ad8dcf2d18bfc9d64d
I20260928 08:31:10.986796 24020 ts_tablet_manager.cc:1939] T 77238a3e549749ad8dcf2d18bfc9d64d P 3fdb1684336f4d56b74b80a45e1dca14: Deleting consensus metadata
I20260928 08:31:10.986886 24021 ts_tablet_manager.cc:1918] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:31:10.987103 24021 log.cc:1196] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-0/wal/wals/77238a3e549749ad8dcf2d18bfc9d64d
I20260928 08:31:10.987773 24021 ts_tablet_manager.cc:1939] T 77238a3e549749ad8dcf2d18bfc9d64d P b215502609054ce4a0ba9def4ab40533: Deleting consensus metadata
I20260928 08:31:10.989104 23473 catalog_manager.cc:4841] TS 3fdb1684336f4d56b74b80a45e1dca14 (127.15.134.195:34143): tablet 77238a3e549749ad8dcf2d18bfc9d64d (table test-table [id=8ba702fd506941b995d77900fe67daa8]) successfully deleted
I20260928 08:31:10.989802 23471 catalog_manager.cc:4841] TS b215502609054ce4a0ba9def4ab40533 (127.15.134.193:44847): tablet 77238a3e549749ad8dcf2d18bfc9d64d (table test-table [id=8ba702fd506941b995d77900fe67daa8]) successfully deleted
I20260928 08:31:11.011335 24016 raft_consensus.cc:482] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:31:11.011622 24016 raft_consensus.cc:504] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.012707 24016 leader_election.cc:288] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cf012777c4f49a3a8d81f43278ac323 (127.15.134.194:34495), 3fdb1684336f4d56b74b80a45e1dca14 (127.15.134.195:34143)
I20260928 08:31:11.013345 23921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec1fd26e43b49969a4d05058080d9a8" candidate_uuid: "b215502609054ce4a0ba9def4ab40533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fdb1684336f4d56b74b80a45e1dca14" is_pre_election: true
I20260928 08:31:11.013388 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec1fd26e43b49969a4d05058080d9a8" candidate_uuid: "b215502609054ce4a0ba9def4ab40533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cf012777c4f49a3a8d81f43278ac323" is_pre_election: true
I20260928 08:31:11.013623 23921 raft_consensus.cc:2454] T 9ec1fd26e43b49969a4d05058080d9a8 P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b215502609054ce4a0ba9def4ab40533 in term 0.
I20260928 08:31:11.013676 23786 raft_consensus.cc:2454] T 9ec1fd26e43b49969a4d05058080d9a8 P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b215502609054ce4a0ba9def4ab40533 in term 0.
I20260928 08:31:11.014210 23576 leader_election.cc:302] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3fdb1684336f4d56b74b80a45e1dca14, 4cf012777c4f49a3a8d81f43278ac323, b215502609054ce4a0ba9def4ab40533; no voters:
I20260928 08:31:11.014554 24016 raft_consensus.cc:2781] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:31:11.014726 24016 raft_consensus.cc:482] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:31:11.014863 24016 raft_consensus.cc:3037] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:11.017269 24016 raft_consensus.cc:504] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.017920 24016 leader_election.cc:288] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533 [CANDIDATE]: Term 1 election: Requested vote from peers 4cf012777c4f49a3a8d81f43278ac323 (127.15.134.194:34495), 3fdb1684336f4d56b74b80a45e1dca14 (127.15.134.195:34143)
I20260928 08:31:11.018433 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec1fd26e43b49969a4d05058080d9a8" candidate_uuid: "b215502609054ce4a0ba9def4ab40533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cf012777c4f49a3a8d81f43278ac323"
I20260928 08:31:11.018546 23921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ec1fd26e43b49969a4d05058080d9a8" candidate_uuid: "b215502609054ce4a0ba9def4ab40533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fdb1684336f4d56b74b80a45e1dca14"
I20260928 08:31:11.018672 23786 raft_consensus.cc:3037] T 9ec1fd26e43b49969a4d05058080d9a8 P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:11.018743 23921 raft_consensus.cc:3037] T 9ec1fd26e43b49969a4d05058080d9a8 P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:11.021737 23786 raft_consensus.cc:2454] T 9ec1fd26e43b49969a4d05058080d9a8 P 4cf012777c4f49a3a8d81f43278ac323 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b215502609054ce4a0ba9def4ab40533 in term 1.
I20260928 08:31:11.021731 23921 raft_consensus.cc:2454] T 9ec1fd26e43b49969a4d05058080d9a8 P 3fdb1684336f4d56b74b80a45e1dca14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b215502609054ce4a0ba9def4ab40533 in term 1.
I20260928 08:31:11.022341 23573 leader_election.cc:302] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cf012777c4f49a3a8d81f43278ac323, b215502609054ce4a0ba9def4ab40533; no voters:
I20260928 08:31:11.022897 24016 raft_consensus.cc:2781] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:31:11.023044 24016 raft_consensus.cc:686] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533 [term 1 LEADER]: Becoming Leader. State: Replica: b215502609054ce4a0ba9def4ab40533, State: Running, Role: LEADER
I20260928 08:31:11.023324 24016 consensus_queue.cc:237] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.026124 23485 catalog_manager.cc:5495] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533 reported cstate change: term changed from 0 to 1, leader changed from <none> to b215502609054ce4a0ba9def4ab40533 (127.15.134.193). New cstate: current_term: 1 leader_uuid: "b215502609054ce4a0ba9def4ab40533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } health_report { overall_health: UNKNOWN } } }
Sep 28 08:31:11 dist-test-slave-qp2n krb5kdc[23441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584268, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.254@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM
I20260928 08:31:11.089013 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:11.067262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:38601 (local address 127.0.0.1:39914)
0928 08:31:11.067986 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:11.068009 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:11.068188 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:11.068845 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:11.068850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:11.069877 (+ 1027us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:11.070253 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:11.070268 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.071739 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.071747 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:11.072842 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:11.072854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.073534 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.073541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:11.073692 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:11.078625 (+ 4933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:11.078672 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:11.081431 (+ 2759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:11.085827 (+ 4396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:11.085839 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:11.085845 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:11.086405 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:11.086847 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:11.086850 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:11.086853 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:11.086985 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:11.087795 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:11.087806 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:11.087994 (+ 188us) client_negotiation.cc:769] Sending connection context
0928 08:31:11.088233 (+ 239us) client_negotiation.cc:240] Negotiation successful
0928 08:31:11.088576 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":382,"threads_started":1}
I20260928 08:31:11.089303 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:11.067274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39914 (local address 127.15.134.254:38601)
0928 08:31:11.067505 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:11.067515 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:11.068123 (+ 608us) server_negotiation.cc:407] Connection header received
0928 08:31:11.068466 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:11.068474 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:11.068551 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:11.068792 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:11.070429 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.071594 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.072979 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.073357 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.073497 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:11.081683 (+ 8186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:11.081697 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:11.081703 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:11.081749 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:11.085584 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:11.086545 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:11.086551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:11.086556 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:11.086722 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:11.087159 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:11.087165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:11.087169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:11.087616 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:11.087806 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:11.088400 (+ 594us) server_negotiation.cc:299] Negotiation successful
0928 08:31:11.088807 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":77}
I20260928 08:31:11.096067 23485 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:39914:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 08:31:11.097558 23485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:31:11.105352 23623 tablet_service.cc:1467] Processing CreateTablet for tablet da56c605dd7246218a02f56abca6f3b7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16ec5aae8c734db6977007a16c58defe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:31:11.105556 23762 tablet_service.cc:1467] Processing CreateTablet for tablet da56c605dd7246218a02f56abca6f3b7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16ec5aae8c734db6977007a16c58defe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:31:11.106037 23623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da56c605dd7246218a02f56abca6f3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:11.106130 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da56c605dd7246218a02f56abca6f3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:11.105993 23901 tablet_service.cc:1467] Processing CreateTablet for tablet da56c605dd7246218a02f56abca6f3b7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=16ec5aae8c734db6977007a16c58defe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:31:11.106487 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da56c605dd7246218a02f56abca6f3b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:11.114042 23996 tablet_bootstrap.cc:492] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323: Bootstrap starting.
I20260928 08:31:11.114948 23997 tablet_bootstrap.cc:492] T da56c605dd7246218a02f56abca6f3b7 P b215502609054ce4a0ba9def4ab40533: Bootstrap starting.
I20260928 08:31:11.115896 23996 tablet_bootstrap.cc:654] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:11.116948 23997 tablet_bootstrap.cc:654] T da56c605dd7246218a02f56abca6f3b7 P b215502609054ce4a0ba9def4ab40533: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:11.118387 23995 tablet_bootstrap.cc:492] T da56c605dd7246218a02f56abca6f3b7 P 3fdb1684336f4d56b74b80a45e1dca14: Bootstrap starting.
I20260928 08:31:11.119854 23996 tablet_bootstrap.cc:492] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323: No bootstrap required, opened a new log
I20260928 08:31:11.120067 23995 tablet_bootstrap.cc:654] T da56c605dd7246218a02f56abca6f3b7 P 3fdb1684336f4d56b74b80a45e1dca14: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:11.120124 23996 ts_tablet_manager.cc:1397] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 08:31:11.121105 23996 raft_consensus.cc:348] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.121397 23996 raft_consensus.cc:374] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:11.121481 23996 raft_consensus.cc:729] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cf012777c4f49a3a8d81f43278ac323, State: Initialized, Role: FOLLOWER
I20260928 08:31:11.121873 23996 consensus_queue.cc:260] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.122700 24003 tablet_replica.cc:439] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } } }
I20260928 08:31:11.123145 24003 tablet_replica.cc:442] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:31:11.127094 23996 ts_tablet_manager.cc:1428] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260928 08:31:11.131999 23995 tablet_bootstrap.cc:492] T da56c605dd7246218a02f56abca6f3b7 P 3fdb1684336f4d56b74b80a45e1dca14: No bootstrap required, opened a new log
I20260928 08:31:11.132212 23997 tablet_bootstrap.cc:492] T da56c605dd7246218a02f56abca6f3b7 P b215502609054ce4a0ba9def4ab40533: No bootstrap required, opened a new log
I20260928 08:31:11.132212 23995 ts_tablet_manager.cc:1397] T da56c605dd7246218a02f56abca6f3b7 P 3fdb1684336f4d56b74b80a45e1dca14: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260928 08:31:11.132427 23997 ts_tablet_manager.cc:1397] T da56c605dd7246218a02f56abca6f3b7 P b215502609054ce4a0ba9def4ab40533: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.007s
I20260928 08:31:11.133335 23997 raft_consensus.cc:348] T da56c605dd7246218a02f56abca6f3b7 P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.133342 23995 raft_consensus.cc:348] T da56c605dd7246218a02f56abca6f3b7 P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.133603 23995 raft_consensus.cc:374] T da56c605dd7246218a02f56abca6f3b7 P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:11.133651 23997 raft_consensus.cc:374] T da56c605dd7246218a02f56abca6f3b7 P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:11.133755 23995 raft_consensus.cc:729] T da56c605dd7246218a02f56abca6f3b7 P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fdb1684336f4d56b74b80a45e1dca14, State: Initialized, Role: FOLLOWER
I20260928 08:31:11.133806 23997 raft_consensus.cc:729] T da56c605dd7246218a02f56abca6f3b7 P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b215502609054ce4a0ba9def4ab40533, State: Initialized, Role: FOLLOWER
I20260928 08:31:11.134141 23997 consensus_queue.cc:260] T da56c605dd7246218a02f56abca6f3b7 P b215502609054ce4a0ba9def4ab40533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.134105 23995 consensus_queue.cc:260] T da56c605dd7246218a02f56abca6f3b7 P 3fdb1684336f4d56b74b80a45e1dca14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.134569 24016 tablet_replica.cc:439] T da56c605dd7246218a02f56abca6f3b7 P b215502609054ce4a0ba9def4ab40533: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } } }
I20260928 08:31:11.134717 23995 ts_tablet_manager.cc:1428] T da56c605dd7246218a02f56abca6f3b7 P 3fdb1684336f4d56b74b80a45e1dca14: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260928 08:31:11.134863 24016 tablet_replica.cc:442] T da56c605dd7246218a02f56abca6f3b7 P b215502609054ce4a0ba9def4ab40533: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:31:11.134665 24002 tablet_replica.cc:439] T da56c605dd7246218a02f56abca6f3b7 P 3fdb1684336f4d56b74b80a45e1dca14: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } } }
I20260928 08:31:11.134980 24002 tablet_replica.cc:442] T da56c605dd7246218a02f56abca6f3b7 P 3fdb1684336f4d56b74b80a45e1dca14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:31:11.136070 23997 ts_tablet_manager.cc:1428] T da56c605dd7246218a02f56abca6f3b7 P b215502609054ce4a0ba9def4ab40533: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 08:31:11.297727 24003 raft_consensus.cc:482] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:31:11.298022 24003 raft_consensus.cc:504] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.300256 24003 leader_election.cc:288] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b215502609054ce4a0ba9def4ab40533 (127.15.134.193:44847), 3fdb1684336f4d56b74b80a45e1dca14 (127.15.134.195:34143)
I20260928 08:31:11.308847 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:11.300848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:35329 (local address 127.15.134.195:34143)
0928 08:31:11.301068 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:11.301078 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:11.301819 (+ 741us) server_negotiation.cc:407] Connection header received
0928 08:31:11.302135 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:11.302140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:11.302213 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:11.302332 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:11.303056 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.304313 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.306677 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.307682 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.307864 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:11.307946 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:11.308324 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 08:31:11.308465 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 08:31:11.309070 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:11.300577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:34143 (local address 127.15.134.194:35329)
0928 08:31:11.301654 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:11.301681 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:11.301875 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:11.302342 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:11.302346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:11.302370 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:11.302824 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:11.302846 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.304455 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.304466 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:11.306557 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:11.306571 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.307901 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.307907 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:11.308075 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:11.308141 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 08:31:11.308302 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 08:31:11.308483 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":916,"mutex_wait_us":19,"thread_start_us":726,"threads_started":1}
I20260928 08:31:11.309738 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:11.300853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:39319 (local address 127.15.134.193:44847)
0928 08:31:11.301094 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:11.301101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:11.301838 (+ 737us) server_negotiation.cc:407] Connection header received
0928 08:31:11.302175 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:11.302182 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:11.302296 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:11.302424 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:11.303206 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.303985 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.306664 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.308218 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.308393 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:11.308474 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:11.309202 (+ 728us) server_negotiation.cc:299] Negotiation successful
0928 08:31:11.309385 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 08:31:11.310153 23921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da56c605dd7246218a02f56abca6f3b7" candidate_uuid: "4cf012777c4f49a3a8d81f43278ac323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fdb1684336f4d56b74b80a45e1dca14" is_pre_election: true
I20260928 08:31:11.310425 23921 raft_consensus.cc:2454] T da56c605dd7246218a02f56abca6f3b7 P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cf012777c4f49a3a8d81f43278ac323 in term 0.
I20260928 08:31:11.310532 24036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:11.300577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:44847 (local address 127.15.134.194:39319)
0928 08:31:11.301715 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:11.301731 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:11.301970 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:11.302523 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:11.302528 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:11.302550 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:11.303040 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:11.303052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.304126 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.304134 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:11.306453 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:11.306474 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.308697 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.308706 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:11.308934 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:11.308983 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 08:31:11.309949 (+ 966us) client_negotiation.cc:240] Negotiation successful
0928 08:31:11.310125 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1003,"mutex_wait_us":270,"thread_start_us":711,"threads_started":1}
I20260928 08:31:11.310969 23716 leader_election.cc:302] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fdb1684336f4d56b74b80a45e1dca14, 4cf012777c4f49a3a8d81f43278ac323; no voters:
I20260928 08:31:11.311448 24003 raft_consensus.cc:2781] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:31:11.311574 24003 raft_consensus.cc:482] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:31:11.311676 24003 raft_consensus.cc:3037] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:11.311453 23643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da56c605dd7246218a02f56abca6f3b7" candidate_uuid: "4cf012777c4f49a3a8d81f43278ac323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b215502609054ce4a0ba9def4ab40533" is_pre_election: true
I20260928 08:31:11.311959 23643 raft_consensus.cc:2454] T da56c605dd7246218a02f56abca6f3b7 P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cf012777c4f49a3a8d81f43278ac323 in term 0.
I20260928 08:31:11.314209 24003 raft_consensus.cc:504] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.314841 24003 leader_election.cc:288] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323 [CANDIDATE]: Term 1 election: Requested vote from peers b215502609054ce4a0ba9def4ab40533 (127.15.134.193:44847), 3fdb1684336f4d56b74b80a45e1dca14 (127.15.134.195:34143)
I20260928 08:31:11.315359 23921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da56c605dd7246218a02f56abca6f3b7" candidate_uuid: "4cf012777c4f49a3a8d81f43278ac323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fdb1684336f4d56b74b80a45e1dca14"
I20260928 08:31:11.315416 23643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da56c605dd7246218a02f56abca6f3b7" candidate_uuid: "4cf012777c4f49a3a8d81f43278ac323" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b215502609054ce4a0ba9def4ab40533"
I20260928 08:31:11.315595 23921 raft_consensus.cc:3037] T da56c605dd7246218a02f56abca6f3b7 P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:11.315626 23643 raft_consensus.cc:3037] T da56c605dd7246218a02f56abca6f3b7 P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:11.318653 23921 raft_consensus.cc:2454] T da56c605dd7246218a02f56abca6f3b7 P 3fdb1684336f4d56b74b80a45e1dca14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cf012777c4f49a3a8d81f43278ac323 in term 1.
I20260928 08:31:11.319008 23643 raft_consensus.cc:2454] T da56c605dd7246218a02f56abca6f3b7 P b215502609054ce4a0ba9def4ab40533 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cf012777c4f49a3a8d81f43278ac323 in term 1.
I20260928 08:31:11.319228 23716 leader_election.cc:302] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fdb1684336f4d56b74b80a45e1dca14, 4cf012777c4f49a3a8d81f43278ac323; no voters:
I20260928 08:31:11.319610 24003 raft_consensus.cc:2781] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:31:11.320657 24003 raft_consensus.cc:686] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323 [term 1 LEADER]: Becoming Leader. State: Replica: 4cf012777c4f49a3a8d81f43278ac323, State: Running, Role: LEADER
I20260928 08:31:11.321130 24003 consensus_queue.cc:237] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.323218 24037 tablet_replica.cc:439] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323: TxnStatusTablet state changed. Reason: New leader 4cf012777c4f49a3a8d81f43278ac323. Latest consensus state: current_term: 1 leader_uuid: "4cf012777c4f49a3a8d81f43278ac323" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } } }
I20260928 08:31:11.323628 24037 tablet_replica.cc:442] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:31:11.324318 24039 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 08:31:11.324548 24039 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 08:31:11.326442 23486 catalog_manager.cc:5495] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cf012777c4f49a3a8d81f43278ac323 (127.15.134.194). New cstate: current_term: 1 leader_uuid: "4cf012777c4f49a3a8d81f43278ac323" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } health_report { overall_health: UNKNOWN } } }
I20260928 08:31:11.381106 24042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:11.368496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37730 (local address 127.15.134.194:34495)
0928 08:31:11.373185 (+ 4689us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:11.373192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:11.373216 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:31:11.373331 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:11.373338 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:11.373457 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:11.373652 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:31:11.374244 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.375563 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.377691 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.378479 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.378726 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:11.379383 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:31:11.380276 (+ 893us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:31:11.380441 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:11.380581 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 08:31:11.380698 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4497,"thread_start_us":4277,"threads_started":1}
I20260928 08:31:11.381392 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:11.368166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:34495 (local address 127.0.0.1:37730)
0928 08:31:11.368423 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:11.368444 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:11.368590 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:11.373723 (+ 5133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:11.373729 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:11.373745 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:31:11.373992 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:11.374006 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.375732 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.375739 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:11.377486 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:11.377505 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.378793 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.378800 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:11.379018 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:11.379106 (+ 88us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:31:11.380412 (+ 1306us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:31:11.380437 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 08:31:11.380537 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 08:31:11.381058 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 08:31:11.386700 23921 raft_consensus.cc:1261] T da56c605dd7246218a02f56abca6f3b7 P 3fdb1684336f4d56b74b80a45e1dca14 [term 1 FOLLOWER]: Refusing update from remote peer 4cf012777c4f49a3a8d81f43278ac323: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:31:11.387494 24003 consensus_queue.cc:1035] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:31:11.387693 23643 raft_consensus.cc:1261] T da56c605dd7246218a02f56abca6f3b7 P b215502609054ce4a0ba9def4ab40533 [term 1 FOLLOWER]: Refusing update from remote peer 4cf012777c4f49a3a8d81f43278ac323: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:31:11.389771 24003 consensus_queue.cc:1035] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:31:11.402189 24016 tablet_replica.cc:439] T da56c605dd7246218a02f56abca6f3b7 P b215502609054ce4a0ba9def4ab40533: TxnStatusTablet state changed. Reason: New leader 4cf012777c4f49a3a8d81f43278ac323. Latest consensus state: current_term: 1 leader_uuid: "4cf012777c4f49a3a8d81f43278ac323" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } } }
I20260928 08:31:11.402535 24016 tablet_replica.cc:442] T da56c605dd7246218a02f56abca6f3b7 P b215502609054ce4a0ba9def4ab40533: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:31:11.403618 24003 tablet_replica.cc:439] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4cf012777c4f49a3a8d81f43278ac323" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } } }
I20260928 08:31:11.403934 24003 tablet_replica.cc:442] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:31:11.403908 24002 tablet_replica.cc:439] T da56c605dd7246218a02f56abca6f3b7 P 3fdb1684336f4d56b74b80a45e1dca14: TxnStatusTablet state changed. Reason: New leader 4cf012777c4f49a3a8d81f43278ac323. Latest consensus state: current_term: 1 leader_uuid: "4cf012777c4f49a3a8d81f43278ac323" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } } }
I20260928 08:31:11.404146 24002 tablet_replica.cc:442] T da56c605dd7246218a02f56abca6f3b7 P 3fdb1684336f4d56b74b80a45e1dca14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:31:11.409440 24012 mvcc.cc:205] Tried to move back new op lower bound from 7334233175591981056 to 7334233175335190528. Current Snapshot: MvccSnapshot[applied={T|T < 7334233175591981056}]
I20260928 08:31:11.410310 24037 tablet_replica.cc:439] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4cf012777c4f49a3a8d81f43278ac323" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } } }
I20260928 08:31:11.410584 24037 tablet_replica.cc:442] T da56c605dd7246218a02f56abca6f3b7 P 4cf012777c4f49a3a8d81f43278ac323: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:31:11.419431 24002 tablet_replica.cc:439] T da56c605dd7246218a02f56abca6f3b7 P 3fdb1684336f4d56b74b80a45e1dca14: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4cf012777c4f49a3a8d81f43278ac323" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } } }
I20260928 08:31:11.419708 24002 tablet_replica.cc:442] T da56c605dd7246218a02f56abca6f3b7 P 3fdb1684336f4d56b74b80a45e1dca14: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:31:11.429118 24016 tablet_replica.cc:439] T da56c605dd7246218a02f56abca6f3b7 P b215502609054ce4a0ba9def4ab40533: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4cf012777c4f49a3a8d81f43278ac323" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } } }
I20260928 08:31:11.429452 24016 tablet_replica.cc:442] T da56c605dd7246218a02f56abca6f3b7 P b215502609054ce4a0ba9def4ab40533: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:31:11.451996 24042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:11.443704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:35581 (local address 127.15.134.194:34495)
0928 08:31:11.443911 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:11.443917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:11.445402 (+ 1485us) server_negotiation.cc:407] Connection header received
0928 08:31:11.445611 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:11.445617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:11.445708 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:11.445812 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:31:11.446455 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.447625 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.449818 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.450452 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.450579 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:11.451140 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:31:11.451282 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:31:11.451388 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:11.451600 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 08:31:11.451688 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 08:31:11.452219 24049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:11.443604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:34495 (local address 127.15.134.193:35581)
0928 08:31:11.445276 (+ 1672us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:11.445297 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:11.445463 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:11.445853 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:11.445859 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:11.445871 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:31:11.446251 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:11.446263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.447769 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.447775 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:11.449656 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:11.449677 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.450672 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.450679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:11.450863 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:11.450942 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:31:11.451408 (+ 466us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:31:11.451427 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 08:31:11.451517 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 08:31:11.451676 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1460,"thread_start_us":1215,"threads_started":1}
I20260928 08:31:11.466423 23921 raft_consensus.cc:1261] T 9ec1fd26e43b49969a4d05058080d9a8 P 3fdb1684336f4d56b74b80a45e1dca14 [term 1 FOLLOWER]: Refusing update from remote peer b215502609054ce4a0ba9def4ab40533: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:31:11.466518 23786 raft_consensus.cc:1261] T 9ec1fd26e43b49969a4d05058080d9a8 P 4cf012777c4f49a3a8d81f43278ac323 [term 1 FOLLOWER]: Refusing update from remote peer b215502609054ce4a0ba9def4ab40533: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:31:11.467188 24016 consensus_queue.cc:1035] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:31:11.467724 24001 consensus_queue.cc:1035] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:31:11.486320 24009 mvcc.cc:205] Tried to move back new op lower bound from 7334233175920746496 to 7334233174114312192. Current Snapshot: MvccSnapshot[applied={T|T < 7334233175920746496}]
I20260928 08:31:11.544891 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:11.534274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:40197 (local address 127.15.134.193:44847)
0928 08:31:11.534487 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:11.534496 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:11.538661 (+ 4165us) server_negotiation.cc:407] Connection header received
0928 08:31:11.538871 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:11.538877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:11.538966 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:11.539075 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:31:11.539834 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.540973 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.542471 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.543147 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.543269 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:11.543986 (+ 717us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:31:11.544124 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:31:11.544185 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:11.544399 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 08:31:11.544521 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 08:31:11.545771 24053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:11.534236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:44847 (local address 127.15.134.194:40197)
0928 08:31:11.538527 (+ 4291us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:11.538550 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:11.538721 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:11.539129 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:11.539133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:11.539146 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:31:11.539631 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:11.539645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.541100 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.541105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:11.542314 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:11.542330 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.543490 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.543498 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:11.543685 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:11.543769 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:31:11.544264 (+ 495us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:31:11.544274 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 08:31:11.545160 (+ 886us) client_negotiation.cc:240] Negotiation successful
0928 08:31:11.545320 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4045,"thread_start_us":3866,"threads_started":1}
I20260928 08:31:11.591527 23486 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:31:11.591769 23486 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:31:11.595124 23486 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1: Sending DeleteTablet for 3 replicas of tablet 9ec1fd26e43b49969a4d05058080d9a8
I20260928 08:31:11.596088 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ec1fd26e43b49969a4d05058080d9a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:31:11 UTC) from {username='oryx', principal='oryx/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:55036
I20260928 08:31:11.600380 23623 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ec1fd26e43b49969a4d05058080d9a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:31:11 UTC) from {username='oryx', principal='oryx/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:58582
I20260928 08:31:11.600829 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ec1fd26e43b49969a4d05058080d9a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:31:11 UTC) from {username='oryx', principal='oryx/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:37728
I20260928 08:31:11.614948 24055 tablet_replica.cc:330] T 9ec1fd26e43b49969a4d05058080d9a8 P 3fdb1684336f4d56b74b80a45e1dca14: stopping tablet replica
I20260928 08:31:11.615445 24055 raft_consensus.cc:2229] T 9ec1fd26e43b49969a4d05058080d9a8 P 3fdb1684336f4d56b74b80a45e1dca14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:31:11.615870 24055 raft_consensus.cc:2258] T 9ec1fd26e43b49969a4d05058080d9a8 P 3fdb1684336f4d56b74b80a45e1dca14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:31:11.616526 24056 tablet_replica.cc:330] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533: stopping tablet replica
I20260928 08:31:11.616822 24056 raft_consensus.cc:2229] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:31:11.617187 24056 raft_consensus.cc:2258] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:31:11.618036 24055 ts_tablet_manager.cc:1905] T 9ec1fd26e43b49969a4d05058080d9a8 P 3fdb1684336f4d56b74b80a45e1dca14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:31:11.619473 24057 tablet_replica.cc:330] T 9ec1fd26e43b49969a4d05058080d9a8 P 4cf012777c4f49a3a8d81f43278ac323: stopping tablet replica
I20260928 08:31:11.619740 24057 raft_consensus.cc:2229] T 9ec1fd26e43b49969a4d05058080d9a8 P 4cf012777c4f49a3a8d81f43278ac323 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:31:11.620074 24057 raft_consensus.cc:2258] T 9ec1fd26e43b49969a4d05058080d9a8 P 4cf012777c4f49a3a8d81f43278ac323 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:31:11.621304 24056 ts_tablet_manager.cc:1905] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:31:11.621304 24057 ts_tablet_manager.cc:1905] T 9ec1fd26e43b49969a4d05058080d9a8 P 4cf012777c4f49a3a8d81f43278ac323: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:31:11.628938 24057 ts_tablet_manager.cc:1918] T 9ec1fd26e43b49969a4d05058080d9a8 P 4cf012777c4f49a3a8d81f43278ac323: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:31:11.629148 24057 log.cc:1196] T 9ec1fd26e43b49969a4d05058080d9a8 P 4cf012777c4f49a3a8d81f43278ac323: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/9ec1fd26e43b49969a4d05058080d9a8
I20260928 08:31:11.629753 24057 ts_tablet_manager.cc:1939] T 9ec1fd26e43b49969a4d05058080d9a8 P 4cf012777c4f49a3a8d81f43278ac323: Deleting consensus metadata
I20260928 08:31:11.631428 23470 catalog_manager.cc:4841] TS 4cf012777c4f49a3a8d81f43278ac323 (127.15.134.194:34495): tablet 9ec1fd26e43b49969a4d05058080d9a8 (table test-table [id=4946a602ecff41e8abb3d39d959e9f96]) successfully deleted
I20260928 08:31:11.628542 24056 ts_tablet_manager.cc:1918] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:31:11.632859 24056 log.cc:1196] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-0/wal/wals/9ec1fd26e43b49969a4d05058080d9a8
I20260928 08:31:11.633391 24056 ts_tablet_manager.cc:1939] T 9ec1fd26e43b49969a4d05058080d9a8 P b215502609054ce4a0ba9def4ab40533: Deleting consensus metadata
I20260928 08:31:11.634649 23471 catalog_manager.cc:4841] TS b215502609054ce4a0ba9def4ab40533 (127.15.134.193:44847): tablet 9ec1fd26e43b49969a4d05058080d9a8 (table test-table [id=4946a602ecff41e8abb3d39d959e9f96]) successfully deleted
I20260928 08:31:11.636020 24055 ts_tablet_manager.cc:1918] T 9ec1fd26e43b49969a4d05058080d9a8 P 3fdb1684336f4d56b74b80a45e1dca14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:31:11.636173 24055 log.cc:1196] T 9ec1fd26e43b49969a4d05058080d9a8 P 3fdb1684336f4d56b74b80a45e1dca14: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/9ec1fd26e43b49969a4d05058080d9a8
I20260928 08:31:11.636588 24055 ts_tablet_manager.cc:1939] T 9ec1fd26e43b49969a4d05058080d9a8 P 3fdb1684336f4d56b74b80a45e1dca14: Deleting consensus metadata
I20260928 08:31:11.637652 23473 catalog_manager.cc:4841] TS 3fdb1684336f4d56b74b80a45e1dca14 (127.15.134.195:34143): tablet 9ec1fd26e43b49969a4d05058080d9a8 (table test-table [id=4946a602ecff41e8abb3d39d959e9f96]) successfully deleted
Sep 28 08:31:11 dist-test-slave-qp2n krb5kdc[23441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 08:31:11 dist-test-slave-qp2n krb5kdc[23441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:11 dist-test-slave-qp2n krb5kdc[23441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 08:31:11.675165 24064 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:11.662427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39922 (local address 127.15.134.254:38601)
0928 08:31:11.663241 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:11.663248 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:11.663534 (+ 286us) server_negotiation.cc:407] Connection header received
0928 08:31:11.663681 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:11.663686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:11.663755 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:11.663858 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:11.665618 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.666400 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.668223 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.668869 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.668978 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:11.674834 (+ 5856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39922: BlockingRecv error: recv got EOF from 127.0.0.1:39922 (error 108)
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":559,"threads_started":1}
Sep 28 08:31:11 dist-test-slave-qp2n krb5kdc[23441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM
I20260928 08:31:11.694815 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:11.678205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39936 (local address 127.15.134.254:38601)
0928 08:31:11.678417 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:11.678424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:11.678852 (+ 428us) server_negotiation.cc:407] Connection header received
0928 08:31:11.679146 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:11.679153 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:11.679249 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:11.679366 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:11.680781 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.682009 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.683470 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.684164 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.684304 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:11.688787 (+ 4483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:11.688794 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:11.688797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:11.688838 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:11.691682 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:11.692572 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:11.692576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:11.692578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:11.692774 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:11.693231 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:11.693234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:11.693236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:11.693523 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:11.693707 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:11.694232 (+ 525us) server_negotiation.cc:299] Negotiation successful
0928 08:31:11.694455 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 08:31:11.699033 23486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:31:11.699720 23486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:31:11.706768 23623 tablet_service.cc:1467] Processing CreateTablet for tablet b802b2a7eede41d486a87e6b7bad05f9 (DEFAULT_TABLE table=test-table [id=c4b8f8066db1402fbec990ac03501ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:31:11.707283 23623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b802b2a7eede41d486a87e6b7bad05f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:11.707173 23762 tablet_service.cc:1467] Processing CreateTablet for tablet b802b2a7eede41d486a87e6b7bad05f9 (DEFAULT_TABLE table=test-table [id=c4b8f8066db1402fbec990ac03501ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:31:11.707625 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b802b2a7eede41d486a87e6b7bad05f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:11.712726 24071 tablet_bootstrap.cc:492] T b802b2a7eede41d486a87e6b7bad05f9 P b215502609054ce4a0ba9def4ab40533: Bootstrap starting.
I20260928 08:31:11.712978 23901 tablet_service.cc:1467] Processing CreateTablet for tablet b802b2a7eede41d486a87e6b7bad05f9 (DEFAULT_TABLE table=test-table [id=c4b8f8066db1402fbec990ac03501ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:31:11.713423 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b802b2a7eede41d486a87e6b7bad05f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:11.714449 24071 tablet_bootstrap.cc:654] T b802b2a7eede41d486a87e6b7bad05f9 P b215502609054ce4a0ba9def4ab40533: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:11.724107 24074 tablet_bootstrap.cc:492] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14: Bootstrap starting.
I20260928 08:31:11.724915 24072 tablet_bootstrap.cc:492] T b802b2a7eede41d486a87e6b7bad05f9 P 4cf012777c4f49a3a8d81f43278ac323: Bootstrap starting.
I20260928 08:31:11.725937 24074 tablet_bootstrap.cc:654] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:11.727222 24072 tablet_bootstrap.cc:654] T b802b2a7eede41d486a87e6b7bad05f9 P 4cf012777c4f49a3a8d81f43278ac323: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:11.729774 24074 tablet_bootstrap.cc:492] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14: No bootstrap required, opened a new log
I20260928 08:31:11.729774 24071 tablet_bootstrap.cc:492] T b802b2a7eede41d486a87e6b7bad05f9 P b215502609054ce4a0ba9def4ab40533: No bootstrap required, opened a new log
I20260928 08:31:11.729938 24071 ts_tablet_manager.cc:1397] T b802b2a7eede41d486a87e6b7bad05f9 P b215502609054ce4a0ba9def4ab40533: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.009s
I20260928 08:31:11.729979 24074 ts_tablet_manager.cc:1397] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 08:31:11.730716 24074 raft_consensus.cc:348] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.730711 24071 raft_consensus.cc:348] T b802b2a7eede41d486a87e6b7bad05f9 P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.730959 24074 raft_consensus.cc:374] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:11.730959 24071 raft_consensus.cc:374] T b802b2a7eede41d486a87e6b7bad05f9 P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:11.731057 24071 raft_consensus.cc:729] T b802b2a7eede41d486a87e6b7bad05f9 P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b215502609054ce4a0ba9def4ab40533, State: Initialized, Role: FOLLOWER
I20260928 08:31:11.731089 24074 raft_consensus.cc:729] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fdb1684336f4d56b74b80a45e1dca14, State: Initialized, Role: FOLLOWER
I20260928 08:31:11.731333 24071 consensus_queue.cc:260] T b802b2a7eede41d486a87e6b7bad05f9 P b215502609054ce4a0ba9def4ab40533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.731354 24074 consensus_queue.cc:260] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.731913 24071 ts_tablet_manager.cc:1428] T b802b2a7eede41d486a87e6b7bad05f9 P b215502609054ce4a0ba9def4ab40533: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 08:31:11.732014 24074 ts_tablet_manager.cc:1428] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:31:11.733532 24072 tablet_bootstrap.cc:492] T b802b2a7eede41d486a87e6b7bad05f9 P 4cf012777c4f49a3a8d81f43278ac323: No bootstrap required, opened a new log
I20260928 08:31:11.733745 24072 ts_tablet_manager.cc:1397] T b802b2a7eede41d486a87e6b7bad05f9 P 4cf012777c4f49a3a8d81f43278ac323: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20260928 08:31:11.734548 24072 raft_consensus.cc:348] T b802b2a7eede41d486a87e6b7bad05f9 P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.734903 24072 raft_consensus.cc:374] T b802b2a7eede41d486a87e6b7bad05f9 P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:11.735057 24072 raft_consensus.cc:729] T b802b2a7eede41d486a87e6b7bad05f9 P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cf012777c4f49a3a8d81f43278ac323, State: Initialized, Role: FOLLOWER
I20260928 08:31:11.735354 24072 consensus_queue.cc:260] T b802b2a7eede41d486a87e6b7bad05f9 P 4cf012777c4f49a3a8d81f43278ac323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.735944 24072 ts_tablet_manager.cc:1428] T b802b2a7eede41d486a87e6b7bad05f9 P 4cf012777c4f49a3a8d81f43278ac323: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 08:31:11.747727 24002 raft_consensus.cc:482] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:31:11.748001 24002 raft_consensus.cc:504] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.750211 24002 leader_election.cc:288] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b215502609054ce4a0ba9def4ab40533 (127.15.134.193:44847), 4cf012777c4f49a3a8d81f43278ac323 (127.15.134.194:34495)
I20260928 08:31:11.760057 24042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:11.750900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:39535 (local address 127.15.134.194:34495)
0928 08:31:11.751191 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:11.751201 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:11.752997 (+ 1796us) server_negotiation.cc:407] Connection header received
0928 08:31:11.753185 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:11.753190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:11.753304 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:11.753406 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:11.754031 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.754948 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.758121 (+ 3173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.758946 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.759155 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:11.759247 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:11.759579 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 08:31:11.759737 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 08:31:11.760506 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:11.750434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:44847 (local address 127.15.134.195:60759)
0928 08:31:11.752534 (+ 2100us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:11.752558 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:11.752811 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:11.753200 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:11.753204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:11.753226 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:11.753603 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:11.753617 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.755182 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.755192 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:11.757864 (+ 2672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:11.757880 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.759602 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.759608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:11.759772 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:11.759802 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 08:31:11.759881 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 08:31:11.759999 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1939,"mutex_wait_us":163,"thread_start_us":1763,"threads_started":1}
I20260928 08:31:11.760892 24078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:11.750507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:34495 (local address 127.15.134.195:39535)
0928 08:31:11.752901 (+ 2394us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:11.752920 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:11.753042 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:11.753428 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:11.753435 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:11.753465 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:11.753883 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:11.753892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.755082 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.755089 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:11.757864 (+ 2775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:11.757880 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.759161 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.759167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:11.759362 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:11.759386 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 08:31:11.760264 (+ 878us) client_negotiation.cc:240] Negotiation successful
0928 08:31:11.760395 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2260,"mutex_wait_us":165,"thread_start_us":2081,"threads_started":1}
I20260928 08:31:11.761586 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b802b2a7eede41d486a87e6b7bad05f9" candidate_uuid: "3fdb1684336f4d56b74b80a45e1dca14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cf012777c4f49a3a8d81f43278ac323" is_pre_election: true
I20260928 08:31:11.761857 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:11.751372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:60759 (local address 127.15.134.193:44847)
0928 08:31:11.751638 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:11.751646 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:11.752677 (+ 1031us) server_negotiation.cc:407] Connection header received
0928 08:31:11.752989 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:11.752995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:11.753072 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:11.753157 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:11.753801 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.754990 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.758043 (+ 3053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.759358 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.761081 (+ 1723us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:11.761359 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:11.761411 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 08:31:11.761554 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 08:31:11.761854 23786 raft_consensus.cc:2454] T b802b2a7eede41d486a87e6b7bad05f9 P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fdb1684336f4d56b74b80a45e1dca14 in term 0.
I20260928 08:31:11.762398 23852 leader_election.cc:302] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fdb1684336f4d56b74b80a45e1dca14, 4cf012777c4f49a3a8d81f43278ac323; no voters:
I20260928 08:31:11.762367 23643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b802b2a7eede41d486a87e6b7bad05f9" candidate_uuid: "3fdb1684336f4d56b74b80a45e1dca14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b215502609054ce4a0ba9def4ab40533" is_pre_election: true
I20260928 08:31:11.762645 23643 raft_consensus.cc:2454] T b802b2a7eede41d486a87e6b7bad05f9 P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fdb1684336f4d56b74b80a45e1dca14 in term 0.
I20260928 08:31:11.762977 24002 raft_consensus.cc:2781] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:31:11.763113 24002 raft_consensus.cc:482] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:31:11.763230 24002 raft_consensus.cc:3037] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:11.766426 24002 raft_consensus.cc:504] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.767179 24002 leader_election.cc:288] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14 [CANDIDATE]: Term 1 election: Requested vote from peers b215502609054ce4a0ba9def4ab40533 (127.15.134.193:44847), 4cf012777c4f49a3a8d81f43278ac323 (127.15.134.194:34495)
I20260928 08:31:11.767604 23643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b802b2a7eede41d486a87e6b7bad05f9" candidate_uuid: "3fdb1684336f4d56b74b80a45e1dca14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b215502609054ce4a0ba9def4ab40533"
I20260928 08:31:11.767822 23643 raft_consensus.cc:3037] T b802b2a7eede41d486a87e6b7bad05f9 P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:11.767778 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b802b2a7eede41d486a87e6b7bad05f9" candidate_uuid: "3fdb1684336f4d56b74b80a45e1dca14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cf012777c4f49a3a8d81f43278ac323"
I20260928 08:31:11.768010 23786 raft_consensus.cc:3037] T b802b2a7eede41d486a87e6b7bad05f9 P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:11.770402 23786 raft_consensus.cc:2454] T b802b2a7eede41d486a87e6b7bad05f9 P 4cf012777c4f49a3a8d81f43278ac323 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fdb1684336f4d56b74b80a45e1dca14 in term 1.
I20260928 08:31:11.770834 23852 leader_election.cc:302] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fdb1684336f4d56b74b80a45e1dca14, 4cf012777c4f49a3a8d81f43278ac323; no voters:
I20260928 08:31:11.770879 23643 raft_consensus.cc:2454] T b802b2a7eede41d486a87e6b7bad05f9 P b215502609054ce4a0ba9def4ab40533 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fdb1684336f4d56b74b80a45e1dca14 in term 1.
I20260928 08:31:11.771122 24002 raft_consensus.cc:2781] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:31:11.771708 24002 raft_consensus.cc:686] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14 [term 1 LEADER]: Becoming Leader. State: Replica: 3fdb1684336f4d56b74b80a45e1dca14, State: Running, Role: LEADER
I20260928 08:31:11.772168 24002 consensus_queue.cc:237] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.775573 23483 catalog_manager.cc:5495] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fdb1684336f4d56b74b80a45e1dca14 (127.15.134.195). New cstate: current_term: 1 leader_uuid: "3fdb1684336f4d56b74b80a45e1dca14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } health_report { overall_health: HEALTHY } } }
I20260928 08:31:11.798314 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:11.791714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55042 (local address 127.15.134.195:34143)
0928 08:31:11.792022 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:11.792027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:11.792047 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:31:11.792283 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:11.792286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:11.792374 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:11.792509 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:31:11.792962 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.793700 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.795753 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.796190 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.796288 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:11.796964 (+ 676us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:31:11.797594 (+ 630us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:31:11.797707 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:11.797860 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 08:31:11.797929 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260928 08:31:11.802034 23643 raft_consensus.cc:1261] T b802b2a7eede41d486a87e6b7bad05f9 P b215502609054ce4a0ba9def4ab40533 [term 1 FOLLOWER]: Refusing update from remote peer 3fdb1684336f4d56b74b80a45e1dca14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:31:11.802103 23786 raft_consensus.cc:1261] T b802b2a7eede41d486a87e6b7bad05f9 P 4cf012777c4f49a3a8d81f43278ac323 [term 1 FOLLOWER]: Refusing update from remote peer 3fdb1684336f4d56b74b80a45e1dca14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:31:11.802814 24002 consensus_queue.cc:1035] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:31:11.803182 24079 consensus_queue.cc:1035] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:31:11.819835 23483 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:31:11.820153 23483 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:31:11.824498 23483 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1: Sending DeleteTablet for 3 replicas of tablet b802b2a7eede41d486a87e6b7bad05f9
I20260928 08:31:11.826011 23623 tablet_service.cc:1514] Processing DeleteTablet for tablet b802b2a7eede41d486a87e6b7bad05f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:31:11 UTC) from {username='oryx', principal='oryx/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:58582
I20260928 08:31:11.826084 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet b802b2a7eede41d486a87e6b7bad05f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:31:11 UTC) from {username='oryx', principal='oryx/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:37728
I20260928 08:31:11.826458 24057 tablet_replica.cc:330] T b802b2a7eede41d486a87e6b7bad05f9 P 4cf012777c4f49a3a8d81f43278ac323: stopping tablet replica
I20260928 08:31:11.826385 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet b802b2a7eede41d486a87e6b7bad05f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:31:11 UTC) from {username='oryx', principal='oryx/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:55036
I20260928 08:31:11.826720 24056 tablet_replica.cc:330] T b802b2a7eede41d486a87e6b7bad05f9 P b215502609054ce4a0ba9def4ab40533: stopping tablet replica
I20260928 08:31:11.826683 24057 raft_consensus.cc:2229] T b802b2a7eede41d486a87e6b7bad05f9 P 4cf012777c4f49a3a8d81f43278ac323 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:31:11.826896 24056 raft_consensus.cc:2229] T b802b2a7eede41d486a87e6b7bad05f9 P b215502609054ce4a0ba9def4ab40533 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:31:11.826949 24057 raft_consensus.cc:2258] T b802b2a7eede41d486a87e6b7bad05f9 P 4cf012777c4f49a3a8d81f43278ac323 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:31:11.827054 24056 pending_rounds.cc:62] T b802b2a7eede41d486a87e6b7bad05f9 P b215502609054ce4a0ba9def4ab40533: Trying to abort 2 pending ops.
I20260928 08:31:11.827183 24056 pending_rounds.cc:68] T b802b2a7eede41d486a87e6b7bad05f9 P b215502609054ce4a0ba9def4ab40533: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7334233177293795328 op_type: WRITE_OP write_request { tablet_id: "b802b2a7eede41d486a87e6b7bad05f9" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\373\315\350\325\006\"8\n\ntest-admin\022*\n c4b8f8066db1402fbec990ac03501ced\020\001\030\001 \001(\001" signature: "<redacted>""\036\316\213F\371\341\033\377pVO\307*\310\305*\262;\235\300\353f\205\035\334\232\331\0164\265W\221\322\206 \020\nCvX%\201c\320C\253H\303>5\377\237\273\232\253M\315\250\371!PR\326\014" signing_key_seq_num: 0 } } request_id { client_id: "9a3a7483a3c1440db2e30355bdc04220" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260928 08:31:11.827967 24056 pending_rounds.cc:68] T b802b2a7eede41d486a87e6b7bad05f9 P b215502609054ce4a0ba9def4ab40533: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7334233177182978048 op_type: NO_OP noop_request { }
I20260928 08:31:11.828118 24056 raft_consensus.cc:2866] T b802b2a7eede41d486a87e6b7bad05f9 P b215502609054ce4a0ba9def4ab40533 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260928 08:31:11.828269 24056 raft_consensus.cc:2258] T b802b2a7eede41d486a87e6b7bad05f9 P b215502609054ce4a0ba9def4ab40533 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:31:11.828316 24057 ts_tablet_manager.cc:1905] T b802b2a7eede41d486a87e6b7bad05f9 P 4cf012777c4f49a3a8d81f43278ac323: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:31:11.828522 24055 tablet_replica.cc:330] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14: stopping tablet replica
I20260928 08:31:11.828766 24055 raft_consensus.cc:2229] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:31:11.829252 24055 raft_consensus.cc:2258] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:31:11.829597 23485 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:31:11.830338 23485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:31:11.830427 24055 ts_tablet_manager.cc:1905] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:31:11.835757 24057 ts_tablet_manager.cc:1918] T b802b2a7eede41d486a87e6b7bad05f9 P 4cf012777c4f49a3a8d81f43278ac323: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:31:11.835968 24057 log.cc:1196] T b802b2a7eede41d486a87e6b7bad05f9 P 4cf012777c4f49a3a8d81f43278ac323: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/b802b2a7eede41d486a87e6b7bad05f9
I20260928 08:31:11.836483 24057 ts_tablet_manager.cc:1939] T b802b2a7eede41d486a87e6b7bad05f9 P 4cf012777c4f49a3a8d81f43278ac323: Deleting consensus metadata
I20260928 08:31:11.837823 23470 catalog_manager.cc:4841] TS 4cf012777c4f49a3a8d81f43278ac323 (127.15.134.194:34495): tablet b802b2a7eede41d486a87e6b7bad05f9 (table test-table [id=c4b8f8066db1402fbec990ac03501ced]) successfully deleted
I20260928 08:31:11.838825 23762 tablet_service.cc:1467] Processing CreateTablet for tablet a26689256af84bfaa4df2624920169fb (DEFAULT_TABLE table=test-table [id=f44ea79f39ab4c2a80af74b7931122c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:31:11.839293 23762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a26689256af84bfaa4df2624920169fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:11.841017 23623 tablet_service.cc:1467] Processing CreateTablet for tablet a26689256af84bfaa4df2624920169fb (DEFAULT_TABLE table=test-table [id=f44ea79f39ab4c2a80af74b7931122c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:31:11.841480 23623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a26689256af84bfaa4df2624920169fb. 1 dirs total, 0 dirs full, 0 dirs failed
W20260928 08:31:11.841900 24082 consensus_queue.cc:1159] T b802b2a7eede41d486a87e6b7bad05f9 P b215502609054ce4a0ba9def4ab40533 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b215502609054ce4a0ba9def4ab40533" status { last_received { term: 1 index: 2 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 2 } }
I20260928 08:31:11.844828 24056 ts_tablet_manager.cc:1905] T b802b2a7eede41d486a87e6b7bad05f9 P b215502609054ce4a0ba9def4ab40533: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:31:11.845544 23901 tablet_service.cc:1467] Processing CreateTablet for tablet a26689256af84bfaa4df2624920169fb (DEFAULT_TABLE table=test-table [id=f44ea79f39ab4c2a80af74b7931122c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:31:11.845970 23901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a26689256af84bfaa4df2624920169fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:11.846372 24072 tablet_bootstrap.cc:492] T a26689256af84bfaa4df2624920169fb P 4cf012777c4f49a3a8d81f43278ac323: Bootstrap starting.
I20260928 08:31:11.848109 24072 tablet_bootstrap.cc:654] T a26689256af84bfaa4df2624920169fb P 4cf012777c4f49a3a8d81f43278ac323: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:11.855528 24074 tablet_bootstrap.cc:492] T a26689256af84bfaa4df2624920169fb P 3fdb1684336f4d56b74b80a45e1dca14: Bootstrap starting.
I20260928 08:31:11.857502 24074 tablet_bootstrap.cc:654] T a26689256af84bfaa4df2624920169fb P 3fdb1684336f4d56b74b80a45e1dca14: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:11.861248 24055 ts_tablet_manager.cc:1918] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:31:11.861395 24071 tablet_bootstrap.cc:492] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533: Bootstrap starting.
I20260928 08:31:11.863077 24071 tablet_bootstrap.cc:654] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:11.865623 24055 log.cc:1196] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/b802b2a7eede41d486a87e6b7bad05f9
I20260928 08:31:11.866294 24056 ts_tablet_manager.cc:1918] T b802b2a7eede41d486a87e6b7bad05f9 P b215502609054ce4a0ba9def4ab40533: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:31:11.866940 24055 ts_tablet_manager.cc:1939] T b802b2a7eede41d486a87e6b7bad05f9 P 3fdb1684336f4d56b74b80a45e1dca14: Deleting consensus metadata
I20260928 08:31:11.867305 24056 log.cc:1196] T b802b2a7eede41d486a87e6b7bad05f9 P b215502609054ce4a0ba9def4ab40533: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-0/wal/wals/b802b2a7eede41d486a87e6b7bad05f9
I20260928 08:31:11.869609 23473 catalog_manager.cc:4841] TS 3fdb1684336f4d56b74b80a45e1dca14 (127.15.134.195:34143): tablet b802b2a7eede41d486a87e6b7bad05f9 (table test-table [id=c4b8f8066db1402fbec990ac03501ced]) successfully deleted
I20260928 08:31:11.869664 24074 tablet_bootstrap.cc:492] T a26689256af84bfaa4df2624920169fb P 3fdb1684336f4d56b74b80a45e1dca14: No bootstrap required, opened a new log
I20260928 08:31:11.869855 24074 ts_tablet_manager.cc:1397] T a26689256af84bfaa4df2624920169fb P 3fdb1684336f4d56b74b80a45e1dca14: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20260928 08:31:11.870760 24074 raft_consensus.cc:348] T a26689256af84bfaa4df2624920169fb P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.871029 24074 raft_consensus.cc:374] T a26689256af84bfaa4df2624920169fb P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:11.871116 24074 raft_consensus.cc:729] T a26689256af84bfaa4df2624920169fb P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fdb1684336f4d56b74b80a45e1dca14, State: Initialized, Role: FOLLOWER
I20260928 08:31:11.871409 24074 consensus_queue.cc:260] T a26689256af84bfaa4df2624920169fb P 3fdb1684336f4d56b74b80a45e1dca14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.871989 24074 ts_tablet_manager.cc:1428] T a26689256af84bfaa4df2624920169fb P 3fdb1684336f4d56b74b80a45e1dca14: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:31:11.872437 24056 ts_tablet_manager.cc:1939] T b802b2a7eede41d486a87e6b7bad05f9 P b215502609054ce4a0ba9def4ab40533: Deleting consensus metadata
I20260928 08:31:11.874418 23471 catalog_manager.cc:4841] TS b215502609054ce4a0ba9def4ab40533 (127.15.134.193:44847): tablet b802b2a7eede41d486a87e6b7bad05f9 (table test-table [id=c4b8f8066db1402fbec990ac03501ced]) successfully deleted
I20260928 08:31:11.878037 24071 tablet_bootstrap.cc:492] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533: No bootstrap required, opened a new log
I20260928 08:31:11.878253 24071 ts_tablet_manager.cc:1397] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.003s
I20260928 08:31:11.879830 24071 raft_consensus.cc:348] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.880085 24071 raft_consensus.cc:374] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:11.880169 24071 raft_consensus.cc:729] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b215502609054ce4a0ba9def4ab40533, State: Initialized, Role: FOLLOWER
I20260928 08:31:11.880426 24071 consensus_queue.cc:260] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.883229 24071 ts_tablet_manager.cc:1428] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260928 08:31:11.884068 24001 raft_consensus.cc:482] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:31:11.884230 24001 raft_consensus.cc:504] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.884821 24001 leader_election.cc:288] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cf012777c4f49a3a8d81f43278ac323 (127.15.134.194:34495), 3fdb1684336f4d56b74b80a45e1dca14 (127.15.134.195:34143)
I20260928 08:31:11.885676 23921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26689256af84bfaa4df2624920169fb" candidate_uuid: "b215502609054ce4a0ba9def4ab40533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fdb1684336f4d56b74b80a45e1dca14" is_pre_election: true
I20260928 08:31:11.885650 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26689256af84bfaa4df2624920169fb" candidate_uuid: "b215502609054ce4a0ba9def4ab40533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cf012777c4f49a3a8d81f43278ac323" is_pre_election: true
I20260928 08:31:11.885917 23921 raft_consensus.cc:2454] T a26689256af84bfaa4df2624920169fb P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b215502609054ce4a0ba9def4ab40533 in term 0.
I20260928 08:31:11.886564 23576 leader_election.cc:302] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fdb1684336f4d56b74b80a45e1dca14, b215502609054ce4a0ba9def4ab40533; no voters:
W20260928 08:31:11.886876 23573 leader_election.cc:341] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4cf012777c4f49a3a8d81f43278ac323 (127.15.134.194:34495): Illegal state: must be running to vote when last-logged opid is not known
I20260928 08:31:11.887375 24001 raft_consensus.cc:2781] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:31:11.887471 24001 raft_consensus.cc:482] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:31:11.887581 24001 raft_consensus.cc:3037] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:11.890012 24072 tablet_bootstrap.cc:492] T a26689256af84bfaa4df2624920169fb P 4cf012777c4f49a3a8d81f43278ac323: No bootstrap required, opened a new log
I20260928 08:31:11.890039 24001 raft_consensus.cc:504] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.890201 24072 ts_tablet_manager.cc:1397] T a26689256af84bfaa4df2624920169fb P 4cf012777c4f49a3a8d81f43278ac323: Time spent bootstrapping tablet: real 0.044s user 0.013s sys 0.000s
I20260928 08:31:11.891084 23786 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26689256af84bfaa4df2624920169fb" candidate_uuid: "b215502609054ce4a0ba9def4ab40533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cf012777c4f49a3a8d81f43278ac323"
I20260928 08:31:11.891258 23921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a26689256af84bfaa4df2624920169fb" candidate_uuid: "b215502609054ce4a0ba9def4ab40533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fdb1684336f4d56b74b80a45e1dca14"
I20260928 08:31:11.891158 24072 raft_consensus.cc:348] T a26689256af84bfaa4df2624920169fb P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.891461 23921 raft_consensus.cc:3037] T a26689256af84bfaa4df2624920169fb P 3fdb1684336f4d56b74b80a45e1dca14 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:11.891496 24072 raft_consensus.cc:374] T a26689256af84bfaa4df2624920169fb P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:11.891578 24072 raft_consensus.cc:729] T a26689256af84bfaa4df2624920169fb P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cf012777c4f49a3a8d81f43278ac323, State: Initialized, Role: FOLLOWER
I20260928 08:31:11.891842 24072 consensus_queue.cc:260] T a26689256af84bfaa4df2624920169fb P 4cf012777c4f49a3a8d81f43278ac323 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.892160 23786 raft_consensus.cc:3037] T a26689256af84bfaa4df2624920169fb P 4cf012777c4f49a3a8d81f43278ac323 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:11.892419 24072 ts_tablet_manager.cc:1428] T a26689256af84bfaa4df2624920169fb P 4cf012777c4f49a3a8d81f43278ac323: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 08:31:11.895042 23921 raft_consensus.cc:2454] T a26689256af84bfaa4df2624920169fb P 3fdb1684336f4d56b74b80a45e1dca14 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b215502609054ce4a0ba9def4ab40533 in term 1.
I20260928 08:31:11.895182 23786 raft_consensus.cc:2454] T a26689256af84bfaa4df2624920169fb P 4cf012777c4f49a3a8d81f43278ac323 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b215502609054ce4a0ba9def4ab40533 in term 1.
I20260928 08:31:11.895756 23573 leader_election.cc:302] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cf012777c4f49a3a8d81f43278ac323, b215502609054ce4a0ba9def4ab40533; no voters:
I20260928 08:31:11.896175 24016 raft_consensus.cc:2781] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:31:11.896363 24001 leader_election.cc:288] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533 [CANDIDATE]: Term 1 election: Requested vote from peers 4cf012777c4f49a3a8d81f43278ac323 (127.15.134.194:34495), 3fdb1684336f4d56b74b80a45e1dca14 (127.15.134.195:34143)
I20260928 08:31:11.907227 24016 raft_consensus.cc:686] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533 [term 1 LEADER]: Becoming Leader. State: Replica: b215502609054ce4a0ba9def4ab40533, State: Running, Role: LEADER
I20260928 08:31:11.907586 24016 consensus_queue.cc:237] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } }
I20260928 08:31:11.910538 23485 catalog_manager.cc:5495] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533 reported cstate change: term changed from 0 to 1, leader changed from <none> to b215502609054ce4a0ba9def4ab40533 (127.15.134.193). New cstate: current_term: 1 leader_uuid: "b215502609054ce4a0ba9def4ab40533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b215502609054ce4a0ba9def4ab40533" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 44847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 } health_report { overall_health: UNKNOWN } } }
I20260928 08:31:11.947913 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:11.942715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58608 (local address 127.15.134.193:44847)
0928 08:31:11.942891 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:11.942897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:11.943085 (+ 188us) server_negotiation.cc:407] Connection header received
0928 08:31:11.943311 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:11.943315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:11.943381 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:11.943476 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:31:11.943871 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.944696 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.946142 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:11.946536 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:11.946676 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:11.947160 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:31:11.947272 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:31:11.947370 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:11.947515 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 08:31:11.947597 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 08:31:11.957365 23786 raft_consensus.cc:1261] T a26689256af84bfaa4df2624920169fb P 4cf012777c4f49a3a8d81f43278ac323 [term 1 FOLLOWER]: Refusing update from remote peer b215502609054ce4a0ba9def4ab40533: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:31:11.957610 23921 raft_consensus.cc:1261] T a26689256af84bfaa4df2624920169fb P 3fdb1684336f4d56b74b80a45e1dca14 [term 1 FOLLOWER]: Refusing update from remote peer b215502609054ce4a0ba9def4ab40533: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:31:11.958039 24016 consensus_queue.cc:1035] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cf012777c4f49a3a8d81f43278ac323" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:31:11.958331 24088 consensus_queue.cc:1035] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fdb1684336f4d56b74b80a45e1dca14" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 34143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:31:12.011737 23485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:31:12.011936 23485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:31:12.014696 23485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 272fa4a356d6455fa214dc28eb720db1: Sending DeleteTablet for 3 replicas of tablet a26689256af84bfaa4df2624920169fb
I20260928 08:31:12.015542 23762 tablet_service.cc:1514] Processing DeleteTablet for tablet a26689256af84bfaa4df2624920169fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:31:12 UTC) from {username='oryx', principal='oryx/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:37728
I20260928 08:31:12.015637 23901 tablet_service.cc:1514] Processing DeleteTablet for tablet a26689256af84bfaa4df2624920169fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:31:12 UTC) from {username='oryx', principal='oryx/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:55036
I20260928 08:31:12.015678 23623 tablet_service.cc:1514] Processing DeleteTablet for tablet a26689256af84bfaa4df2624920169fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:31:12 UTC) from {username='oryx', principal='oryx/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:58582
I20260928 08:31:12.015962 24056 tablet_replica.cc:330] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533: stopping tablet replica
I20260928 08:31:12.015965 24057 tablet_replica.cc:330] T a26689256af84bfaa4df2624920169fb P 4cf012777c4f49a3a8d81f43278ac323: stopping tablet replica
I20260928 08:31:12.015974 24055 tablet_replica.cc:330] T a26689256af84bfaa4df2624920169fb P 3fdb1684336f4d56b74b80a45e1dca14: stopping tablet replica
I20260928 08:31:12.016140 24057 raft_consensus.cc:2229] T a26689256af84bfaa4df2624920169fb P 4cf012777c4f49a3a8d81f43278ac323 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:31:12.016148 24056 raft_consensus.cc:2229] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:31:12.016153 24055 raft_consensus.cc:2229] T a26689256af84bfaa4df2624920169fb P 3fdb1684336f4d56b74b80a45e1dca14 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:31:12.016331 24057 raft_consensus.cc:2258] T a26689256af84bfaa4df2624920169fb P 4cf012777c4f49a3a8d81f43278ac323 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:31:12.016341 24055 raft_consensus.cc:2258] T a26689256af84bfaa4df2624920169fb P 3fdb1684336f4d56b74b80a45e1dca14 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:31:12.016450 24056 raft_consensus.cc:2258] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:31:12.017529 24056 ts_tablet_manager.cc:1905] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:31:12.017536 24057 ts_tablet_manager.cc:1905] T a26689256af84bfaa4df2624920169fb P 4cf012777c4f49a3a8d81f43278ac323: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:31:12.018105 24055 ts_tablet_manager.cc:1905] T a26689256af84bfaa4df2624920169fb P 3fdb1684336f4d56b74b80a45e1dca14: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:31:12.024037 24057 ts_tablet_manager.cc:1918] T a26689256af84bfaa4df2624920169fb P 4cf012777c4f49a3a8d81f43278ac323: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:31:12.024186 24057 log.cc:1196] T a26689256af84bfaa4df2624920169fb P 4cf012777c4f49a3a8d81f43278ac323: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/a26689256af84bfaa4df2624920169fb
I20260928 08:31:12.024364 24056 ts_tablet_manager.cc:1918] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:31:12.024497 24055 ts_tablet_manager.cc:1918] T a26689256af84bfaa4df2624920169fb P 3fdb1684336f4d56b74b80a45e1dca14: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:31:12.024523 24056 log.cc:1196] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-0/wal/wals/a26689256af84bfaa4df2624920169fb
I20260928 08:31:12.024667 24055 log.cc:1196] T a26689256af84bfaa4df2624920169fb P 3fdb1684336f4d56b74b80a45e1dca14: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/a26689256af84bfaa4df2624920169fb
I20260928 08:31:12.024701 24057 ts_tablet_manager.cc:1939] T a26689256af84bfaa4df2624920169fb P 4cf012777c4f49a3a8d81f43278ac323: Deleting consensus metadata
I20260928 08:31:12.025074 24055 ts_tablet_manager.cc:1939] T a26689256af84bfaa4df2624920169fb P 3fdb1684336f4d56b74b80a45e1dca14: Deleting consensus metadata
I20260928 08:31:12.025097 24056 ts_tablet_manager.cc:1939] T a26689256af84bfaa4df2624920169fb P b215502609054ce4a0ba9def4ab40533: Deleting consensus metadata
I20260928 08:31:12.026074 23470 catalog_manager.cc:4841] TS 4cf012777c4f49a3a8d81f43278ac323 (127.15.134.194:34495): tablet a26689256af84bfaa4df2624920169fb (table test-table [id=f44ea79f39ab4c2a80af74b7931122c4]) successfully deleted
I20260928 08:31:12.026094 23473 catalog_manager.cc:4841] TS 3fdb1684336f4d56b74b80a45e1dca14 (127.15.134.195:34143): tablet a26689256af84bfaa4df2624920169fb (table test-table [id=f44ea79f39ab4c2a80af74b7931122c4]) successfully deleted
I20260928 08:31:12.026439 23471 catalog_manager.cc:4841] TS b215502609054ce4a0ba9def4ab40533 (127.15.134.193:44847): tablet a26689256af84bfaa4df2624920169fb (table test-table [id=f44ea79f39ab4c2a80af74b7931122c4]) successfully deleted
I20260928 08:31:12.028927 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 23557
Sep 28 08:31:12 dist-test-slave-qp2n krb5kdc[23441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.134.193@KRBTEST.COM
I20260928 08:31:12.050040 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:12.030112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58620 (local address 127.15.134.193:44847)
0928 08:31:12.030291 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:12.030297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:12.030658 (+ 361us) server_negotiation.cc:407] Connection header received
0928 08:31:12.030819 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:12.030824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:12.030908 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:12.031017 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:12.032469 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:12.033642 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:12.035477 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:12.036161 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:12.036288 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:12.041938 (+ 5650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:12.041960 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:12.041982 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:12.042084 (+ 102us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:12.046147 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:12.047170 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:12.047182 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:12.047186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:12.047353 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:12.047779 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:12.047784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:12.047788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:12.048326 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:12.048661 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:12.049224 (+ 563us) server_negotiation.cc:299] Negotiation successful
0928 08:31:12.049469 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 08:31:12.050705 23688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:12.180416 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 23557
I20260928 08:31:12.205468 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 23697
Sep 28 08:31:12 dist-test-slave-qp2n krb5kdc[23441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.134.194@KRBTEST.COM
I20260928 08:31:12.222339 24042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:12.206320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37746 (local address 127.15.134.194:34495)
0928 08:31:12.206505 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:12.206512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:12.206622 (+ 110us) server_negotiation.cc:407] Connection header received
0928 08:31:12.206824 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:12.206830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:12.206914 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:12.207024 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:12.208300 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:12.209111 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:12.210348 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:12.210773 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:12.210906 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:12.215404 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:12.215424 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:12.215447 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:12.215505 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:12.219339 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:12.220040 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:12.220050 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:12.220054 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:12.220235 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:12.220598 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:12.220603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:12.220673 (+ 70us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:12.221172 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:12.221476 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:12.221722 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 08:31:12.221966 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:31:12.223102 23827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:12.346949 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 23697
I20260928 08:31:12.370231 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 23836
Sep 28 08:31:12 dist-test-slave-qp2n krb5kdc[23441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.15.134.195@KRBTEST.COM
I20260928 08:31:12.390413 24095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:12.371126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55054 (local address 127.15.134.195:34143)
0928 08:31:12.371605 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:12.371609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:12.371626 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:31:12.371681 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:12.371685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:12.371744 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:12.371846 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:12.373201 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:12.373915 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:12.375141 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:12.375598 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:12.375695 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:12.384467 (+ 8772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:12.384484 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:12.384503 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:12.384551 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:12.387281 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:12.388064 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:12.388074 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:12.388077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:12.388233 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:12.388727 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:12.388734 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:12.388740 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:12.389256 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:12.389559 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:12.389807 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 08:31:12.389985 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":238,"threads_started":1}
I20260928 08:31:12.391175 23966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:12.538606 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 23836
I20260928 08:31:12.561652 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 23454
I20260928 08:31:12.562922 23526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:12.683706 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 23454
2026-09-28T08:31:12Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5654 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:31:12 dist-test-slave-qp2n krb5kdc[24100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:31:12 dist-test-slave-qp2n krb5kdc[24100](info): set up 2 sockets
Sep 28 08:31:12 dist-test-slave-qp2n krb5kdc[24100](info): commencing operation
krb5kdc: starting...
W20260928 08:31:13.799234 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:31:13 dist-test-slave-qp2n krb5kdc[24100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:31:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:31:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:31:14.028170 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:33273
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:33273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:31:14.365350 24113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:14.365687 24113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:14.365777 24113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:14.373191 24113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:31:14.373313 24113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:14.373361 24113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:14.373420 24113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:31:14.373472 24113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:31:14.383638 24113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:33273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:33273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:14.384991 24113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:14.386282 24113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:14.399305 24118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:14.399345 24121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:14.399375 24119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:14.400241 24113 server_base.cc:1034] running on GCE node
I20260928 08:31:14.401221 24113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:14.402885 24113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:14.404160 24113 hybrid_clock.cc:648] HybridClock initialized: now 1790584274404085 us; error 63 us; skew 500 ppm
I20260928 08:31:14.404740 24113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T08:31:14Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1792 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:31:14 dist-test-slave-qp2n krb5kdc[24127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:31:14 dist-test-slave-qp2n krb5kdc[24127](info): set up 2 sockets
Sep 28 08:31:14 dist-test-slave-qp2n krb5kdc[24127](info): commencing operation
krb5kdc: starting...
W20260928 08:31:15.620026 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:31:15 dist-test-slave-qp2n krb5kdc[24127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:31:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:31:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:31:15.850184 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:39675
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:39675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:31:16.185590 24140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:16.185946 24140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:16.186125 24140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:16.193329 24140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:31:16.193436 24140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:16.193559 24140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:16.193660 24140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:31:16.193743 24140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:31:16.203672 24140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:39675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:39675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:16.205062 24140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:16.206398 24140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:16.218763 24145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:16.218683 24146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:16.218722 24148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:16.220037 24140 server_base.cc:1034] running on GCE node
I20260928 08:31:16.220958 24140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:16.222527 24140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:16.223773 24140 hybrid_clock.cc:648] HybridClock initialized: now 1790584276223728 us; error 55 us; skew 500 ppm
I20260928 08:31:16.224148 24140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:31:16 dist-test-slave-qp2n krb5kdc[24127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584276, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:31:16.231673 24140 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:31:16.233407 24140 webserver.cc:460] Webserver started at http://127.15.134.254:35571/ using document root <none> and password file <none>
I20260928 08:31:16.234143 24140 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:16.234293 24140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:16.234617 24140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:31:16.237767 24140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "b66bb5a9c8884710ad5020283ac92a27"
format_stamp: "Formatted at 2026-09-28 08:31:16 on dist-test-slave-qp2n"
server_key: "53fff5a02f2449a6bb0eed7797c26c40"
server_key_iv: "1c8bf285410def834c8098d8e976d41e"
server_key_version: "encryptionkey@0"
I20260928 08:31:16.238582 24140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "b66bb5a9c8884710ad5020283ac92a27"
format_stamp: "Formatted at 2026-09-28 08:31:16 on dist-test-slave-qp2n"
server_key: "53fff5a02f2449a6bb0eed7797c26c40"
server_key_iv: "1c8bf285410def834c8098d8e976d41e"
server_key_version: "encryptionkey@0"
I20260928 08:31:16.244454 24140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260928 08:31:16.248210 24155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:16.250010 24140 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:31:16.250231 24140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "b66bb5a9c8884710ad5020283ac92a27"
format_stamp: "Formatted at 2026-09-28 08:31:16 on dist-test-slave-qp2n"
server_key: "53fff5a02f2449a6bb0eed7797c26c40"
server_key_iv: "1c8bf285410def834c8098d8e976d41e"
server_key_version: "encryptionkey@0"
I20260928 08:31:16.250451 24140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:16.282418 24140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:16.286588 24140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:16.286885 24140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:16.305498 24140 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:39675
I20260928 08:31:16.305540 24206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:39675 every 8 connection(s)
I20260928 08:31:16.306845 24140 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:31:16.310492 24207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:16.313162 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 24140
I20260928 08:31:16.313387 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:31:16.314224 15899 external_mini_cluster.cc:1428] Setting key 79d5df8a050e638c9124c75dbde8466a
I20260928 08:31:16.322302 24207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27: Bootstrap starting.
I20260928 08:31:16.325336 24207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:16.326332 24207 log.cc:824] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27: Log is configured to *not* fsync() on all Append() calls
Sep 28 08:31:16 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:31:16.329311 24207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27: No bootstrap required, opened a new log
I20260928 08:31:16.336491 24210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:16.316430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34786 (local address 127.15.134.254:39675)
0928 08:31:16.317192 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:16.317203 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:16.317243 (+ 40us) server_negotiation.cc:407] Connection header received
0928 08:31:16.318868 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:16.318889 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:16.319261 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:16.319650 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:16.321165 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:16.322538 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:16.323755 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:16.324403 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:16.324592 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:16.329009 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:16.329017 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:16.329031 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:16.329068 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:16.332178 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:16.332982 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:16.332989 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:16.333001 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:16.333165 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:16.333555 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:16.333559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:16.333561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:16.334095 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:16.334453 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:16.334922 (+ 469us) server_negotiation.cc:299] Negotiation successful
0928 08:31:16.335372 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":342,"threads_started":1}
I20260928 08:31:16.337642 24207 raft_consensus.cc:348] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66bb5a9c8884710ad5020283ac92a27" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 39675 } }
I20260928 08:31:16.338017 24207 raft_consensus.cc:374] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:16.338122 24207 raft_consensus.cc:729] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66bb5a9c8884710ad5020283ac92a27, State: Initialized, Role: FOLLOWER
I20260928 08:31:16.338707 24207 consensus_queue.cc:260] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66bb5a9c8884710ad5020283ac92a27" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 39675 } }
I20260928 08:31:16.338956 24207 raft_consensus.cc:388] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:31:16.339087 24207 raft_consensus.cc:482] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:31:16.339277 24207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:16.341598 24207 raft_consensus.cc:504] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66bb5a9c8884710ad5020283ac92a27" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 39675 } }
I20260928 08:31:16.342149 24207 leader_election.cc:302] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b66bb5a9c8884710ad5020283ac92a27; no voters:
I20260928 08:31:16.342767 24207 leader_election.cc:288] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:31:16.342916 24212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:31:16.343597 24212 raft_consensus.cc:686] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [term 1 LEADER]: Becoming Leader. State: Replica: b66bb5a9c8884710ad5020283ac92a27, State: Running, Role: LEADER
I20260928 08:31:16.344151 24212 consensus_queue.cc:237] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66bb5a9c8884710ad5020283ac92a27" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 39675 } }
I20260928 08:31:16.345146 24207 sys_catalog.cc:564] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:31:16.347278 24214 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b66bb5a9c8884710ad5020283ac92a27. Latest consensus state: current_term: 1 leader_uuid: "b66bb5a9c8884710ad5020283ac92a27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66bb5a9c8884710ad5020283ac92a27" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 39675 } } }
I20260928 08:31:16.347251 24213 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b66bb5a9c8884710ad5020283ac92a27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66bb5a9c8884710ad5020283ac92a27" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 39675 } } }
I20260928 08:31:16.347719 24213 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [sys.catalog]: This master's current role is: LEADER
I20260928 08:31:16.347719 24214 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [sys.catalog]: This master's current role is: LEADER
I20260928 08:31:16.349786 24218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:31:16.355249 24218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:31:16.366441 24218 catalog_manager.cc:1347] Generated new cluster ID: 6f65a8b4a2f040648672e5b182a6a6cc
I20260928 08:31:16.366576 24218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:31:16.381695 24218 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:31:16.383075 24218 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:31:16.390257 24218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27: Generated new TSK 0
I20260928 08:31:16.391189 24218 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.134.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
I20260928 08:31:16.452986 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:0
--local_ip_for_outbound_sockets=127.15.134.193
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:39675
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:31:16.798729 24233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:16.799031 24233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:16.799217 24233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:16.806572 24233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:16.806685 24233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:16.806881 24233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:31:16.817701 24233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:39675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:16.819096 24233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:16.820426 24233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:16.833724 24241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:16.833819 24239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:16.833966 24238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:16.835029 24233 server_base.cc:1034] running on GCE node
I20260928 08:31:16.835659 24233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:16.836683 24233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:16.837985 24233 hybrid_clock.cc:648] HybridClock initialized: now 1790584276837838 us; error 119 us; skew 500 ppm
I20260928 08:31:16.838337 24233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:31:16 dist-test-slave-qp2n krb5kdc[24127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584276, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:31:16.841537 24233 init.cc:377] Logged in from keytab as kudu/127.15.134.193@KRBTEST.COM (short username kudu)
I20260928 08:31:16.843066 24233 webserver.cc:460] Webserver started at http://127.15.134.193:46491/ using document root <none> and password file <none>
I20260928 08:31:16.843752 24233 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:16.843895 24233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:16.844213 24233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:31:16.847416 24233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/data/instance:
uuid: "d2d3441dd656464cac33ecd635707666"
format_stamp: "Formatted at 2026-09-28 08:31:16 on dist-test-slave-qp2n"
server_key: "4942a80fd7e67afab8e451b569b15251"
server_key_iv: "d97d7bb6b02ad6505157da0cc144e817"
server_key_version: "encryptionkey@0"
I20260928 08:31:16.848277 24233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance:
uuid: "d2d3441dd656464cac33ecd635707666"
format_stamp: "Formatted at 2026-09-28 08:31:16 on dist-test-slave-qp2n"
server_key: "4942a80fd7e67afab8e451b569b15251"
server_key_iv: "d97d7bb6b02ad6505157da0cc144e817"
server_key_version: "encryptionkey@0"
I20260928 08:31:16.854230 24233 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 08:31:16.858560 24248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:16.860146 24233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:31:16.860368 24233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "d2d3441dd656464cac33ecd635707666"
format_stamp: "Formatted at 2026-09-28 08:31:16 on dist-test-slave-qp2n"
server_key: "4942a80fd7e67afab8e451b569b15251"
server_key_iv: "d97d7bb6b02ad6505157da0cc144e817"
server_key_version: "encryptionkey@0"
I20260928 08:31:16.860618 24233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:16.885707 24233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:16.890419 24233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:16.890724 24233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:16.891687 24233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:31:16.893132 24233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:31:16.893234 24233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:16.893376 24233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:31:16.893450 24233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:16.927438 24233 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:35663
I20260928 08:31:16.927526 24360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:35663 every 8 connection(s)
I20260928 08:31:16.928740 24233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:31:16.938890 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 24233
I20260928 08:31:16.939148 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance
I20260928 08:31:16.939889 15899 external_mini_cluster.cc:1428] Setting key 63688225fdcc50d092ce7b9f439b787b
Sep 28 08:31:16 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584276, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:31:16.948750 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:16.931211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:44765 (local address 127.15.134.254:39675)
0928 08:31:16.931681 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:16.931685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:16.932457 (+ 772us) server_negotiation.cc:407] Connection header received
0928 08:31:16.934374 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:16.934379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:16.934470 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:16.934590 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:16.936043 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:16.936869 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:16.938450 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:16.938870 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:16.939022 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:16.942960 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:16.942973 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:16.942976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:16.943007 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:16.945716 (+ 2709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:16.946441 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:16.946446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:16.946453 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:16.946594 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:16.947036 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:16.947041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:16.947045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:16.947309 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:16.947468 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:16.948155 (+ 687us) server_negotiation.cc:299] Negotiation successful
0928 08:31:16.948370 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":248,"threads_started":1}
I20260928 08:31:16.949754 24364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:16.931608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.193:44765)
0928 08:31:16.932294 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:16.932331 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:16.934098 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:16.934842 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:16.934850 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:16.935322 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:16.935874 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:16.935890 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:16.937029 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:16.937034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:16.938303 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:16.938316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:16.939125 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:16.939133 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:16.939412 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:16.940165 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:16.940187 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:16.942742 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:16.945864 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:16.945872 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:16.945890 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:16.946323 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:16.946705 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:16.946710 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:16.946714 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:16.946897 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:16.947452 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:16.947460 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:16.947785 (+ 325us) client_negotiation.cc:769] Sending connection context
0928 08:31:16.948051 (+ 266us) client_negotiation.cc:240] Negotiation successful
0928 08:31:16.948594 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":263,"threads_started":1}
I20260928 08:31:16.951282 24361 heartbeater.cc:347] Connected to a master server at 127.15.134.254:39675
I20260928 08:31:16.951622 24361 heartbeater.cc:464] Registering TS with master...
I20260928 08:31:16.952453 24361 heartbeater.cc:511] Master 127.15.134.254:39675 requested a full tablet report, sending...
I20260928 08:31:16.954944 24172 ts_manager.cc:194] Registered new tserver with Master: d2d3441dd656464cac33ecd635707666 (127.15.134.193:35663)
I20260928 08:31:16.957110 24172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.193@KRBTEST.COM'} at 127.15.134.193:44765
WARNING: no policy specified for kudu/127.15.134.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
I20260928 08:31:16.993185 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:0
--local_ip_for_outbound_sockets=127.15.134.194
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:39675
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:31:17.320000 24367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:17.320256 24367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:17.320431 24367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:17.327585 24367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:17.327687 24367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:17.327894 24367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:31:17.337551 24367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:39675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:17.338730 24367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:17.340001 24367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:17.353130 24373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:17.353344 24372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:17.353150 24375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:17.354115 24367 server_base.cc:1034] running on GCE node
I20260928 08:31:17.354815 24367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:17.355901 24367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:17.357129 24367 hybrid_clock.cc:648] HybridClock initialized: now 1790584277357090 us; error 38 us; skew 500 ppm
I20260928 08:31:17.357460 24367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:31:17 dist-test-slave-qp2n krb5kdc[24127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584277, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:31:17.360772 24367 init.cc:377] Logged in from keytab as kudu/127.15.134.194@KRBTEST.COM (short username kudu)
I20260928 08:31:17.362370 24367 webserver.cc:460] Webserver started at http://127.15.134.194:36633/ using document root <none> and password file <none>
I20260928 08:31:17.363101 24367 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:17.363245 24367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:17.363560 24367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:31:17.366724 24367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/data/instance:
uuid: "f7e72a90c7ef42619d3fc81635eec7fb"
format_stamp: "Formatted at 2026-09-28 08:31:17 on dist-test-slave-qp2n"
server_key: "24208b4b90c2a2aea132eeef81001e17"
server_key_iv: "7f266ade58447926f03532d46faa8e2f"
server_key_version: "encryptionkey@0"
I20260928 08:31:17.367527 24367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance:
uuid: "f7e72a90c7ef42619d3fc81635eec7fb"
format_stamp: "Formatted at 2026-09-28 08:31:17 on dist-test-slave-qp2n"
server_key: "24208b4b90c2a2aea132eeef81001e17"
server_key_iv: "7f266ade58447926f03532d46faa8e2f"
server_key_version: "encryptionkey@0"
I20260928 08:31:17.372848 24367 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 08:31:17.376569 24382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:17.378108 24367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:31:17.378449 24367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "f7e72a90c7ef42619d3fc81635eec7fb"
format_stamp: "Formatted at 2026-09-28 08:31:17 on dist-test-slave-qp2n"
server_key: "24208b4b90c2a2aea132eeef81001e17"
server_key_iv: "7f266ade58447926f03532d46faa8e2f"
server_key_version: "encryptionkey@0"
I20260928 08:31:17.378690 24367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:17.398275 24367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:17.403184 24367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:17.403537 24367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:17.404706 24367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:31:17.406119 24367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:31:17.406224 24367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:17.406401 24367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:31:17.406476 24367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:17.440956 24367 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:38821
I20260928 08:31:17.441043 24494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:38821 every 8 connection(s)
I20260928 08:31:17.442358 24367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:31:17.445740 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 24367
I20260928 08:31:17.445946 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance
I20260928 08:31:17.446516 15899 external_mini_cluster.cc:1428] Setting key 0e0aa161bae888848b18c4c5ab2a343d
Sep 28 08:31:17 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584277, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:31:17.466153 24363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:17.445017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:33187 (local address 127.15.134.254:39675)
0928 08:31:17.445195 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:17.445199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:17.446711 (+ 1512us) server_negotiation.cc:407] Connection header received
0928 08:31:17.449197 (+ 2486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:17.449201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:17.449271 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:17.449357 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:17.451742 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:17.452487 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:17.455096 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:17.455607 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:17.455709 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:17.459733 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:17.459739 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:17.459742 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:17.459777 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:17.462534 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:17.463299 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:17.463304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:17.463307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:17.463432 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:17.463908 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:17.463912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:17.463914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:17.464257 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:17.464508 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:17.465382 (+ 874us) server_negotiation.cc:299] Negotiation successful
0928 08:31:17.465688 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:31:17.467077 24497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:17.445499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.194:33187)
0928 08:31:17.446517 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:17.446567 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:17.448726 (+ 2159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:17.449687 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:17.449700 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:17.450583 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:17.451538 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:17.451566 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:17.452677 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:17.452685 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:17.454926 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:17.454944 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:17.455838 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:17.455847 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:17.456206 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:17.457217 (+ 1011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:17.457240 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:17.459471 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:17.462693 (+ 3222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:17.462703 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:17.462721 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:17.463155 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:17.463546 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:17.463551 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:17.463555 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:17.463757 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:17.464477 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:17.464486 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:17.464943 (+ 457us) client_negotiation.cc:769] Sending connection context
0928 08:31:17.465240 (+ 297us) client_negotiation.cc:240] Negotiation successful
0928 08:31:17.465727 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":468,"threads_started":1}
I20260928 08:31:17.468719 24495 heartbeater.cc:347] Connected to a master server at 127.15.134.254:39675
I20260928 08:31:17.469046 24495 heartbeater.cc:464] Registering TS with master...
I20260928 08:31:17.469800 24495 heartbeater.cc:511] Master 127.15.134.254:39675 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.134.195@KRBTEST.COM; defaulting to no policy
I20260928 08:31:17.471447 24172 ts_manager.cc:194] Registered new tserver with Master: f7e72a90c7ef42619d3fc81635eec7fb (127.15.134.194:38821)
I20260928 08:31:17.472363 24172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.194@KRBTEST.COM'} at 127.15.134.194:33187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
I20260928 08:31:17.500306 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:0
--local_ip_for_outbound_sockets=127.15.134.195
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:39675
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:31:17.849707 24500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:17.850031 24500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:17.850201 24500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:17.857973 24500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:17.858098 24500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:17.858299 24500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:31:17.868848 24500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:39675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:17.870252 24500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:17.871675 24500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:17.885584 24505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:17.885659 24506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:17.885632 24508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:17.886652 24500 server_base.cc:1034] running on GCE node
I20260928 08:31:17.887298 24500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:17.888319 24500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:17.889555 24500 hybrid_clock.cc:648] HybridClock initialized: now 1790584277889498 us; error 101 us; skew 500 ppm
I20260928 08:31:17.889905 24500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:31:17 dist-test-slave-qp2n krb5kdc[24127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584277, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:31:17.893270 24500 init.cc:377] Logged in from keytab as kudu/127.15.134.195@KRBTEST.COM (short username kudu)
I20260928 08:31:17.894879 24500 webserver.cc:460] Webserver started at http://127.15.134.195:42331/ using document root <none> and password file <none>
I20260928 08:31:17.895624 24500 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:17.895773 24500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:17.896099 24500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:31:17.899204 24500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/data/instance:
uuid: "9c8fcc14e1b347ce90cc66bc70c47e73"
format_stamp: "Formatted at 2026-09-28 08:31:17 on dist-test-slave-qp2n"
server_key: "02fbc0a2dbf3a633652e3228c32579aa"
server_key_iv: "31288f3300686891cd6e2a63f0fdeba8"
server_key_version: "encryptionkey@0"
I20260928 08:31:17.900040 24500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance:
uuid: "9c8fcc14e1b347ce90cc66bc70c47e73"
format_stamp: "Formatted at 2026-09-28 08:31:17 on dist-test-slave-qp2n"
server_key: "02fbc0a2dbf3a633652e3228c32579aa"
server_key_iv: "31288f3300686891cd6e2a63f0fdeba8"
server_key_version: "encryptionkey@0"
I20260928 08:31:17.905527 24500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 08:31:17.909180 24515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:17.910459 24500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 08:31:17.910635 24500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "9c8fcc14e1b347ce90cc66bc70c47e73"
format_stamp: "Formatted at 2026-09-28 08:31:17 on dist-test-slave-qp2n"
server_key: "02fbc0a2dbf3a633652e3228c32579aa"
server_key_iv: "31288f3300686891cd6e2a63f0fdeba8"
server_key_version: "encryptionkey@0"
I20260928 08:31:17.910843 24500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:17.930929 24500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:17.935532 24500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:17.935849 24500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:17.936926 24500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:31:17.938323 24500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:31:17.938433 24500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:17.938592 24500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:31:17.938666 24500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:17.961118 24361 heartbeater.cc:503] Master 127.15.134.254:39675 was elected leader, sending a full tablet report...
I20260928 08:31:17.975739 24500 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:42727
I20260928 08:31:17.975917 24627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:42727 every 8 connection(s)
I20260928 08:31:17.977129 24500 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:31:17.983263 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 24500
I20260928 08:31:17.983474 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance
I20260928 08:31:17.984040 15899 external_mini_cluster.cc:1428] Setting key 28d1ea88f1d98c194f041802e90f5380
Sep 28 08:31:17 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584277, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:31:17.999536 24630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:17.979653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:53999 (local address 127.15.134.254:39675)
0928 08:31:17.980113 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:17.980118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:17.981324 (+ 1206us) server_negotiation.cc:407] Connection header received
0928 08:31:17.983253 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:17.983259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:17.983357 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:17.983490 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:17.985629 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:17.986562 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:17.988356 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:17.988831 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:17.988968 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:17.992971 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:17.992977 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:17.992981 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:17.993019 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:17.995862 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:17.996694 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:17.996698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:17.996700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:17.996815 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:17.997374 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:17.997380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:17.997383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:17.997768 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:17.997959 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:17.998894 (+ 935us) server_negotiation.cc:299] Negotiation successful
0928 08:31:17.999124 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":233,"threads_started":1}
I20260928 08:31:18.000569 24631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:17.980230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.195:53999)
0928 08:31:17.981122 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:17.981200 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:17.982982 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:17.983790 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:17.983802 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:17.984690 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:17.985446 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:17.985473 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:17.986729 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:17.986735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:17.988193 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:17.988206 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:17.989066 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:17.989072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:17.989311 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:17.990107 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:17.990137 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:17.992595 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:17.996029 (+ 3434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:17.996038 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:17.996052 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:17.996496 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:17.997023 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:17.997027 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:17.997029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:17.997214 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:17.997938 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:17.997947 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:17.998457 (+ 510us) client_negotiation.cc:769] Sending connection context
0928 08:31:17.998750 (+ 293us) client_negotiation.cc:240] Negotiation successful
0928 08:31:17.999330 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":330,"threads_started":1}
I20260928 08:31:18.002046 24628 heartbeater.cc:347] Connected to a master server at 127.15.134.254:39675
I20260928 08:31:18.002375 24628 heartbeater.cc:464] Registering TS with master...
I20260928 08:31:18.003153 24628 heartbeater.cc:511] Master 127.15.134.254:39675 requested a full tablet report, sending...
I20260928 08:31:18.004658 24172 ts_manager.cc:194] Registered new tserver with Master: 9c8fcc14e1b347ce90cc66bc70c47e73 (127.15.134.195:42727)
I20260928 08:31:18.005465 24172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.195@KRBTEST.COM'} at 127.15.134.195:53999
I20260928 08:31:18.012437 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.15.134.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.134.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.134.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.193.keytab.
Entry for principal oryx/127.15.134.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.193.keytab.
WARNING: no policy specified for oryx/127.15.134.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.134.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.134.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.194.keytab.
Entry for principal oryx/127.15.134.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.194.keytab.
WARNING: no policy specified for oryx/127.15.134.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.15.134.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.15.134.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.195.keytab.
Entry for principal oryx/127.15.134.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.195.keytab.
I20260928 08:31:18.156540 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 24233
Sep 28 08:31:18 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.193@KRBTEST.COM
I20260928 08:31:18.174237 24639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:18.157985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36134 (local address 127.15.134.193:35663)
0928 08:31:18.158784 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:18.158796 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:18.158835 (+ 39us) server_negotiation.cc:407] Connection header received
0928 08:31:18.158926 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:18.158935 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:18.159313 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:18.159511 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:18.160961 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:18.162067 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:18.163465 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:18.164119 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:18.164222 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:18.168577 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:18.168589 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:18.168605 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:18.168696 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:18.171614 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:18.172301 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:18.172308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:18.172311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:18.172436 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:18.172843 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:18.172847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:18.172849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:18.173209 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:18.173434 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:18.173716 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 08:31:18.173911 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":503,"threads_started":1}
I20260928 08:31:18.175514 24356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:18.261312 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 24233
I20260928 08:31:18.279120 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 24367
Sep 28 08:31:18 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.194@KRBTEST.COM
I20260928 08:31:18.304127 24641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:18.280233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36832 (local address 127.15.134.194:38821)
0928 08:31:18.288178 (+ 7945us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:18.288189 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:18.288228 (+ 39us) server_negotiation.cc:407] Connection header received
0928 08:31:18.288326 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:18.288333 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:18.288567 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:18.288828 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:18.290281 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:18.291311 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:18.292675 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:18.293248 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:18.293410 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:18.297830 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:18.297842 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:18.297859 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:18.297899 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:18.301129 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:18.301890 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:18.301898 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:18.301901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:18.302038 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:18.302438 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:18.302447 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:18.302450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:18.302833 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:18.303152 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:18.303479 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 08:31:18.303759 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7708,"thread_start_us":7561,"threads_started":1}
I20260928 08:31:18.305550 24490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:18.403640 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 24367
I20260928 08:31:18.424800 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 24500
Sep 28 08:31:18 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.195@KRBTEST.COM
I20260928 08:31:18.441658 24643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:18.426045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.15.134.195:42727)
0928 08:31:18.426609 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:18.426616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:18.426646 (+ 30us) server_negotiation.cc:407] Connection header received
0928 08:31:18.426709 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:18.426716 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:18.426938 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:18.427124 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:18.428464 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:18.429536 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:18.430780 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:18.431337 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:18.431482 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:18.435986 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:18.435998 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:18.436013 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:18.436054 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:18.438870 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:18.439593 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:18.439602 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:18.439604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:18.439735 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:18.440104 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:18.440108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:18.440110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:18.440462 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:18.440756 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:18.441148 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 08:31:18.441359 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":273,"threads_started":1}
I20260928 08:31:18.443189 24623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:18.527856 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 24500
I20260928 08:31:18.545291 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 24140
I20260928 08:31:18.546643 24202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:18.654753 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 24140
I20260928 08:31:18.676800 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:39675
--webserver_interface=127.15.134.254
--webserver_port=35571
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:39675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:31:19.012395 24646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:19.012755 24646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:19.012939 24646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:19.020676 24646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:31:19.020792 24646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:19.020887 24646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:19.020995 24646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:31:19.021059 24646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:31:19.031040 24646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:39675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:39675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=35571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:19.032444 24646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:19.033896 24646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:19.046432 24652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:19.046458 24654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:19.046432 24651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:19.047225 24646 server_base.cc:1034] running on GCE node
I20260928 08:31:19.048164 24646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:19.049803 24646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:19.051086 24646 hybrid_clock.cc:648] HybridClock initialized: now 1790584279051033 us; error 62 us; skew 500 ppm
I20260928 08:31:19.051518 24646 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:31:19 dist-test-slave-qp2n krb5kdc[24127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584279, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:31:19.054972 24646 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:31:19.056598 24646 webserver.cc:460] Webserver started at http://127.15.134.254:35571/ using document root <none> and password file <none>
I20260928 08:31:19.057423 24646 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:19.057586 24646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:19.063407 24646 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260928 08:31:19.066634 24661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:19.068267 24646 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 08:31:19.068451 24646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "b66bb5a9c8884710ad5020283ac92a27"
format_stamp: "Formatted at 2026-09-28 08:31:16 on dist-test-slave-qp2n"
server_key: "53fff5a02f2449a6bb0eed7797c26c40"
server_key_iv: "1c8bf285410def834c8098d8e976d41e"
server_key_version: "encryptionkey@0"
I20260928 08:31:19.069546 24646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:19.087005 24646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:19.092233 24646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:19.092559 24646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:19.113570 24646 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:39675
I20260928 08:31:19.113613 24712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:39675 every 8 connection(s)
I20260928 08:31:19.115266 24646 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:31:19.119864 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 24646
I20260928 08:31:19.120858 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:35663
--local_ip_for_outbound_sockets=127.15.134.193
--tserver_master_addrs=127.15.134.254:39675
--webserver_port=46491
--webserver_interface=127.15.134.193
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.134.193
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.193.keytab with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:31:19.123481 24713 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:31:19.128774 24713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27: Bootstrap starting.
I20260928 08:31:19.148051 24713 log.cc:824] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27: Log is configured to *not* fsync() on all Append() calls
I20260928 08:31:19.157078 24713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:31:19.157620 24713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27: Bootstrap complete.
I20260928 08:31:19.166810 24713 raft_consensus.cc:348] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66bb5a9c8884710ad5020283ac92a27" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 39675 } }
I20260928 08:31:19.167666 24713 raft_consensus.cc:729] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66bb5a9c8884710ad5020283ac92a27, State: Initialized, Role: FOLLOWER
I20260928 08:31:19.168419 24713 consensus_queue.cc:260] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66bb5a9c8884710ad5020283ac92a27" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 39675 } }
I20260928 08:31:19.168682 24713 raft_consensus.cc:388] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:31:19.168821 24713 raft_consensus.cc:482] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:31:19.169018 24713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [term 1 FOLLOWER]: Advancing to term 2
I20260928 08:31:19.172923 24713 raft_consensus.cc:504] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66bb5a9c8884710ad5020283ac92a27" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 39675 } }
I20260928 08:31:19.173512 24713 leader_election.cc:302] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b66bb5a9c8884710ad5020283ac92a27; no voters:
I20260928 08:31:19.174160 24713 leader_election.cc:288] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 08:31:19.174490 24717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 08:31:19.175253 24717 raft_consensus.cc:686] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [term 2 LEADER]: Becoming Leader. State: Replica: b66bb5a9c8884710ad5020283ac92a27, State: Running, Role: LEADER
I20260928 08:31:19.175944 24717 consensus_queue.cc:237] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66bb5a9c8884710ad5020283ac92a27" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 39675 } }
I20260928 08:31:19.177893 24717 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b66bb5a9c8884710ad5020283ac92a27. Latest consensus state: current_term: 2 leader_uuid: "b66bb5a9c8884710ad5020283ac92a27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66bb5a9c8884710ad5020283ac92a27" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 39675 } } }
I20260928 08:31:19.178218 24717 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [sys.catalog]: This master's current role is: LEADER
I20260928 08:31:19.178857 24719 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b66bb5a9c8884710ad5020283ac92a27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66bb5a9c8884710ad5020283ac92a27" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 39675 } } }
I20260928 08:31:19.179212 24719 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [sys.catalog]: This master's current role is: LEADER
I20260928 08:31:19.182494 24713 sys_catalog.cc:564] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:31:19.183045 24721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:31:19.189684 24721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:31:19.192245 24721 catalog_manager.cc:1259] Loaded cluster ID: 6f65a8b4a2f040648672e5b182a6a6cc
I20260928 08:31:19.192351 24721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:31:19.197415 24721 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:31:19.199307 24721 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b66bb5a9c8884710ad5020283ac92a27: Loaded TSK: 0
I20260928 08:31:19.201263 24721 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 08:31:19.476140 24715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:19.476464 24715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:19.476683 24715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:19.484393 24715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:19.484555 24715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:19.484750 24715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:31:19.495240 24715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.193.keytab
--principal=oryx/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:35663
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=46491
--tserver_master_addrs=127.15.134.254:39675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:19.496752 24715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:19.498107 24715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:19.513561 24740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:19.513598 24739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:19.515249 24742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:19.516042 24715 server_base.cc:1034] running on GCE node
I20260928 08:31:19.516855 24715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:19.518052 24715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:19.519300 24715 hybrid_clock.cc:648] HybridClock initialized: now 1790584279519241 us; error 54 us; skew 500 ppm
I20260928 08:31:19.519652 24715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:31:19 dist-test-slave-qp2n krb5kdc[24127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584279, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:31:19.523133 24715 init.cc:377] Logged in from keytab as oryx/127.15.134.193@KRBTEST.COM (short username oryx)
I20260928 08:31:19.524930 24715 webserver.cc:460] Webserver started at http://127.15.134.193:46491/ using document root <none> and password file <none>
I20260928 08:31:19.525710 24715 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:19.525858 24715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:19.531459 24715 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 08:31:19.534371 24749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:19.535884 24715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:31:19.536041 24715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "d2d3441dd656464cac33ecd635707666"
format_stamp: "Formatted at 2026-09-28 08:31:16 on dist-test-slave-qp2n"
server_key: "4942a80fd7e67afab8e451b569b15251"
server_key_iv: "d97d7bb6b02ad6505157da0cc144e817"
server_key_version: "encryptionkey@0"
I20260928 08:31:19.537088 24715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:19.578737 24715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:19.583382 24715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:19.583663 24715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:19.584748 24715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:31:19.586222 24715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:31:19.586315 24715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:19.586462 24715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:31:19.586553 24715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:19.620532 24715 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:35663
I20260928 08:31:19.620657 24861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:35663 every 8 connection(s)
I20260928 08:31:19.622263 24715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:31:19.627502 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 24715
I20260928 08:31:19.628288 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:38821
--local_ip_for_outbound_sockets=127.15.134.194
--tserver_master_addrs=127.15.134.254:39675
--webserver_port=36633
--webserver_interface=127.15.134.194
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.134.194
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.194.keytab with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:31:19 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:19 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:19.643393 24865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:19.625716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.193:57849)
0928 08:31:19.626606 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:19.626651 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:19.629288 (+ 2637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:19.631456 (+ 2168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:19.631467 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:19.632380 (+ 913us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:19.633056 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:19.633075 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:19.634673 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:19.634679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:19.635939 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:19.635951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:19.637126 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:19.637133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:19.637418 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:19.638140 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:19.638163 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:19.642046 (+ 3883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":401,"threads_started":1}
W20260928 08:31:19.643422 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:19.625526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:57849 (local address 127.15.134.254:39675)
0928 08:31:19.626391 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:19.626409 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:19.626791 (+ 382us) server_negotiation.cc:407] Connection header received
0928 08:31:19.630287 (+ 3496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:19.630308 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:19.630835 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:19.631297 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:19.633262 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:19.634499 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:19.636098 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:19.636884 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:19.637086 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:19.642415 (+ 5329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.193:57849: BlockingRecv error: recv got EOF from 127.15.134.193:57849 (error 108)
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":351,"threads_started":1}
W20260928 08:31:19.644284 24862 heartbeater.cc:644] Failed to heartbeat to 127.15.134.254:39675 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.134.254:39675: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Sep 28 08:31:19 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:19 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:19.653163 24865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:19.644945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.193:51163)
0928 08:31:19.645102 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:19.645125 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:19.645250 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:19.645601 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:19.645605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:19.645960 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:19.646099 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:19.646107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:19.647167 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:19.647172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:19.648091 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:19.648102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:19.648895 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:19.648901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:19.649085 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:19.649753 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:19.649778 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:19.652860 (+ 3082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 08:31:19.653456 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:19.645122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:51163 (local address 127.15.134.254:39675)
0928 08:31:19.645313 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:19.645319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:19.645337 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:31:19.645392 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:19.645397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:19.645467 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:19.645572 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:19.646238 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:19.647015 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:19.648240 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:19.648729 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:19.648880 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:19.653078 (+ 4198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.193:51163: BlockingRecv error: recv got EOF from 127.15.134.193:51163 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 08:31:19 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:19 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:19.663513 24865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:19.654145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.193:57363)
0928 08:31:19.654437 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:19.654456 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:19.654589 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:19.654925 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:19.654930 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:19.655326 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:19.655556 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:19.655568 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:19.656588 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:19.656594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:19.658257 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:19.658273 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:19.658942 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:19.658948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:19.659154 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:19.659781 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:19.659804 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:19.663171 (+ 3367us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166}
W20260928 08:31:19.663666 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:19.654232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:57363 (local address 127.15.134.254:39675)
0928 08:31:19.654407 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:19.654418 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:19.654538 (+ 120us) server_negotiation.cc:407] Connection header received
0928 08:31:19.654713 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:19.654717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:19.654793 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:19.654872 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:19.655703 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:19.656434 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:19.658423 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:19.658803 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:19.658897 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:19.663389 (+ 4492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.193:57363: BlockingRecv error: recv got EOF from 127.15.134.193:57363 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 08:31:19.664142 24862 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:31:19.970177 24866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:19.970534 24866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:19.970633 24866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:19.977788 24866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:19.977896 24866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:19.978004 24866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:31:19.987725 24866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.194.keytab
--principal=oryx/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:38821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=36633
--tserver_master_addrs=127.15.134.254:39675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:19.988991 24866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:19.990211 24866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:20.003154 24871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:20.003219 24872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:20.003415 24874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:20.003932 24866 server_base.cc:1034] running on GCE node
I20260928 08:31:20.004521 24866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:20.005502 24866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:20.006736 24866 hybrid_clock.cc:648] HybridClock initialized: now 1790584280006690 us; error 39 us; skew 500 ppm
I20260928 08:31:20.007058 24866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:31:20 dist-test-slave-qp2n krb5kdc[24127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584280, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:31:20.010123 24866 init.cc:377] Logged in from keytab as oryx/127.15.134.194@KRBTEST.COM (short username oryx)
I20260928 08:31:20.011641 24866 webserver.cc:460] Webserver started at http://127.15.134.194:36633/ using document root <none> and password file <none>
I20260928 08:31:20.012333 24866 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:20.012485 24866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:20.018107 24866 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 08:31:20.021181 24881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:20.022665 24866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:31:20.022868 24866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "f7e72a90c7ef42619d3fc81635eec7fb"
format_stamp: "Formatted at 2026-09-28 08:31:17 on dist-test-slave-qp2n"
server_key: "24208b4b90c2a2aea132eeef81001e17"
server_key_iv: "7f266ade58447926f03532d46faa8e2f"
server_key_version: "encryptionkey@0"
I20260928 08:31:20.023885 24866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:20.047477 24866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:20.052069 24866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:20.052407 24866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:20.053498 24866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:31:20.054971 24866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:31:20.055109 24866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:20.055251 24866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:31:20.055342 24866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:20.090248 24866 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:38821
I20260928 08:31:20.090340 24993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:38821 every 8 connection(s)
I20260928 08:31:20.091733 24866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:31:20.093216 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 24866
I20260928 08:31:20.094075 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:42727
--local_ip_for_outbound_sockets=127.15.134.195
--tserver_master_addrs=127.15.134.254:39675
--webserver_port=42331
--webserver_interface=127.15.134.195
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.15.134.195
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.195.keytab with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:31:20 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:20 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 08:31:20.110879 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:20.095038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:39367 (local address 127.15.134.254:39675)
0928 08:31:20.095297 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:20.095307 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:20.096586 (+ 1279us) server_negotiation.cc:407] Connection header received
0928 08:31:20.099598 (+ 3012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:20.099603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:20.099673 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:20.099776 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:20.102161 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.102924 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.104633 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.105085 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.105243 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:20.110568 (+ 5325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.194:39367: BlockingRecv error: recv got EOF from 127.15.134.194:39367 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 08:31:20.111620 24996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:20.095523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.194:39367)
0928 08:31:20.096376 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:20.096430 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:20.099248 (+ 2818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:20.100147 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:20.100163 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:20.101034 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:20.101933 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:20.101965 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.103107 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.103113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:20.104432 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:20.104445 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.105363 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.105371 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:20.105641 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:20.106429 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:20.106466 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:20.110360 (+ 3894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":301,"threads_started":1}
W20260928 08:31:20.112340 24994 heartbeater.cc:644] Failed to heartbeat to 127.15.134.254:39675 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.134.254:39675: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Sep 28 08:31:20 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:20 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:20.123443 24996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:20.113063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.194:54367)
0928 08:31:20.113354 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:20.113376 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:20.113494 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:20.113884 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:20.113890 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:20.114378 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:20.114601 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:20.114617 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.116044 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.116052 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:20.117861 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:20.117880 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.118884 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.118889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:20.119099 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:20.119749 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:20.119774 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:20.123157 (+ 3383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148}
W20260928 08:31:20.123646 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:20.113243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:54367 (local address 127.15.134.254:39675)
0928 08:31:20.113459 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:20.113465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:20.113485 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:31:20.113634 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:20.113641 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:20.113726 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:20.113823 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:20.114767 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.115889 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.118042 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.118701 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.118833 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:20.123405 (+ 4572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.194:54367: BlockingRecv error: recv got EOF from 127.15.134.194:54367 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 28 08:31:20 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:20 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:20.133324 24996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:20.124340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.194:50681)
0928 08:31:20.124499 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:20.124517 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:20.124638 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:20.125362 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:20.125366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:20.125724 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:20.125887 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:20.125895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.127409 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.127412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:20.128450 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:20.128464 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.129399 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.129402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:20.129525 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:20.130095 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:20.130111 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:20.133104 (+ 2993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 08:31:20.133569 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:20.124563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:50681 (local address 127.15.134.254:39675)
0928 08:31:20.125064 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:20.125070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:20.125086 (+ 16us) server_negotiation.cc:407] Connection header received
0928 08:31:20.125136 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:20.125141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:20.125209 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:20.125308 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:20.126020 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.127141 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.128597 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.129263 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.129384 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:20.133339 (+ 3955us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.194:50681: BlockingRecv error: recv got EOF from 127.15.134.194:50681 (error 108)
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":252,"threads_started":1}
W20260928 08:31:20.133699 24994 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:31:20.442018 24997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:20.442441 24997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:20.442606 24997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:20.451171 24997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:20.451282 24997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:20.451396 24997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:31:20.461505 24997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/krb5kdc/oryx_127.15.134.195.keytab
--principal=oryx/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:42727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=42331
--tserver_master_addrs=127.15.134.254:39675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:20.462790 24997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:20.464102 24997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:20.477792 25003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:20.477722 25004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:20.478268 25006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:20.478826 24997 server_base.cc:1034] running on GCE node
I20260928 08:31:20.479480 24997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:20.480518 24997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:20.481765 24997 hybrid_clock.cc:648] HybridClock initialized: now 1790584280481670 us; error 90 us; skew 500 ppm
I20260928 08:31:20.482168 24997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:31:20 dist-test-slave-qp2n krb5kdc[24127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584280, etypes {rep=17 tkt=17 ses=17}, oryx/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:31:20.485821 24997 init.cc:377] Logged in from keytab as oryx/127.15.134.195@KRBTEST.COM (short username oryx)
I20260928 08:31:20.487658 24997 webserver.cc:460] Webserver started at http://127.15.134.195:42331/ using document root <none> and password file <none>
I20260928 08:31:20.488557 24997 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:20.488754 24997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:20.494889 24997 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 08:31:20.498242 25013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:20.499835 24997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:31:20.500037 24997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "9c8fcc14e1b347ce90cc66bc70c47e73"
format_stamp: "Formatted at 2026-09-28 08:31:17 on dist-test-slave-qp2n"
server_key: "02fbc0a2dbf3a633652e3228c32579aa"
server_key_iv: "31288f3300686891cd6e2a63f0fdeba8"
server_key_version: "encryptionkey@0"
I20260928 08:31:20.501140 24997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:20.529091 24997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:20.533201 24997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:20.533502 24997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:20.534524 24997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:31:20.535908 24997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:31:20.536043 24997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:20.536180 24997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:31:20.536273 24997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:20.571480 24997 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:42727
I20260928 08:31:20.571571 25125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:42727 every 8 connection(s)
I20260928 08:31:20.573071 24997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:31:20.579984 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 24997
Sep 28 08:31:20 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:20 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 08:31:20.590447 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:20.575923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:37399 (local address 127.15.134.254:39675)
0928 08:31:20.576116 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:20.576120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:20.577379 (+ 1259us) server_negotiation.cc:407] Connection header received
0928 08:31:20.579251 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:20.579255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:20.579366 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:20.579457 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:20.581606 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.582360 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.583962 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.584409 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.584547 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:20.590145 (+ 5598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.195:37399: BlockingRecv error: recv got EOF from 127.15.134.195:37399 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 08:31:20.591416 25128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:20.576497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.195:37399)
0928 08:31:20.577215 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:20.577253 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:20.578979 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:20.579667 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:20.579675 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:20.580302 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:20.581413 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:20.581440 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.582529 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.582539 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:20.583784 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:20.583796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.584670 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.584686 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:20.584948 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:20.585625 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:20.585648 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:20.589952 (+ 4304us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":270,"threads_started":1}
W20260928 08:31:20.592141 25126 heartbeater.cc:644] Failed to heartbeat to 127.15.134.254:39675 (0 consecutive failures): Not authorized: Failed to ping master at 127.15.134.254:39675: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
I20260928 08:31:20.597457 24998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:20.580995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32894 (local address 127.15.134.254:39675)
0928 08:31:20.581157 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:20.581165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:20.581568 (+ 403us) server_negotiation.cc:407] Connection header received
0928 08:31:20.581773 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:20.581779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:20.581852 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:20.581965 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:20.583656 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.584585 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.586046 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.586453 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.586621 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:20.589553 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:20.589564 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:20.589586 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:20.589629 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:20.593974 (+ 4345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:20.594691 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:20.594699 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:20.594705 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:20.594916 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:20.595296 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:20.595309 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:20.595313 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:20.595774 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:20.596128 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:20.596630 (+ 502us) server_negotiation.cc:299] Negotiation successful
0928 08:31:20.597019 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 08:31:20 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:20 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:20.600723 25128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:20.592800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.195:36915)
0928 08:31:20.593101 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:20.593121 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:20.593227 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:20.593552 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:20.593557 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:20.593898 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:20.594030 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:20.594038 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.594982 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.594987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:20.595887 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:20.595898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.596694 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.596699 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:20.596881 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:20.597548 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:20.597569 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:20.600394 (+ 2825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170}
W20260928 08:31:20.600867 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:20.592866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:36915 (local address 127.15.134.254:39675)
0928 08:31:20.593117 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:20.593142 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:20.593164 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:31:20.593344 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:20.593349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:20.593428 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:20.593529 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:20.594168 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.594834 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.596026 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.596513 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.596689 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:20.600572 (+ 3883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.195:36915: BlockingRecv error: recv got EOF from 127.15.134.195:36915 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 28 08:31:20 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:20 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:20.609463 25128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:20.601652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.195:46313)
0928 08:31:20.601801 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:20.601815 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:20.602031 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:20.602359 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:20.602364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:20.602691 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:20.602828 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:20.602836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.603822 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.603829 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:20.604856 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:20.604868 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.605552 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.605557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:20.605765 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:20.606351 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:20.606368 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:20.609153 (+ 2785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 08:31:20.609651 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:20.601742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:46313 (local address 127.15.134.254:39675)
0928 08:31:20.601970 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:20.601975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:20.601993 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:31:20.602162 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:20.602168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:20.602241 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:20.602336 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:20.602973 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.603688 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.605003 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.605411 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.605543 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:20.609344 (+ 3801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.195:46313: BlockingRecv error: recv got EOF from 127.15.134.195:46313 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
W20260928 08:31:20.609791 25126 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 08:31:20 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:20 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:20.674037 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:20.664821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.193:37791)
0928 08:31:20.665411 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:20.665429 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:20.665561 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:20.665889 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:20.665892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:20.666416 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:20.666622 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:20.666632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.667661 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.667665 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:20.668804 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:20.668817 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.669545 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.669550 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:20.669680 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:20.670400 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:20.670419 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:20.673606 (+ 3187us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":313,"threads_started":1}
W20260928 08:31:20.674177 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:20.664935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:37791 (local address 127.15.134.254:39675)
0928 08:31:20.665119 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:20.665123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:20.665516 (+ 393us) server_negotiation.cc:407] Connection header received
0928 08:31:20.665663 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:20.665667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:20.665730 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:20.665818 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:20.666760 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.667521 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.668944 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:20.669367 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:20.669461 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:20.673875 (+ 4414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.193:37791: BlockingRecv error: recv got EOF from 127.15.134.193:37791 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 08:31:21 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:21 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:21.143749 25131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:21.134449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.194:41335)
0928 08:31:21.135009 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:21.135025 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:21.135145 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:21.135449 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:21.135452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:21.135959 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:21.136158 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:21.136168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:21.137213 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:21.137217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:21.138240 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:21.138253 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:21.138987 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:21.138992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:21.139115 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:21.139802 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:21.139820 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:21.143404 (+ 3584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":297,"threads_started":1}
W20260928 08:31:21.143900 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:21.134546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:41335 (local address 127.15.134.254:39675)
0928 08:31:21.134742 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:21.134747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:21.135106 (+ 359us) server_negotiation.cc:407] Connection header received
0928 08:31:21.135253 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:21.135256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:21.135324 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:21.135420 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:21.136291 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:21.137072 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:21.138389 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:21.138818 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:21.138927 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:21.143644 (+ 4717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.194:41335: BlockingRecv error: recv got EOF from 127.15.134.194:41335 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 08:31:21 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:21 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:21.622503 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:21.610467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.195:37423)
0928 08:31:21.611013 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:21.611028 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:21.611181 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:21.611580 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:21.611584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:21.612139 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:21.612356 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:21.612370 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:21.613559 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:21.613563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:21.614515 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:21.614526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:21.615237 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:21.615241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:21.615355 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:21.619119 (+ 3764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:21.619137 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:21.622278 (+ 3141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":294,"threads_started":1}
W20260928 08:31:21.622776 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:21.610567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:37423 (local address 127.15.134.254:39675)
0928 08:31:21.610886 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:21.610892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:21.611117 (+ 225us) server_negotiation.cc:407] Connection header received
0928 08:31:21.611356 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:21.611361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:21.611425 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:21.611512 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:21.612511 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:21.613406 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:21.614658 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:21.615083 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:21.615185 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:21.622521 (+ 7336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.195:37423: BlockingRecv error: recv got EOF from 127.15.134.195:37423 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 28 08:31:21 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:21 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:21.684253 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:21.675135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.193:40577)
0928 08:31:21.675669 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:21.675686 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:21.676034 (+ 348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:21.676454 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:21.676457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:21.677007 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:21.677189 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:21.677198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:21.678446 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:21.678450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:21.679383 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:21.679396 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:21.680160 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:21.680164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:21.680285 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:21.680966 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:21.680984 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:21.683960 (+ 2976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":305,"threads_started":1}
W20260928 08:31:21.684751 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:21.675199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:40577 (local address 127.15.134.254:39675)
0928 08:31:21.675392 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:21.675398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:21.675791 (+ 393us) server_negotiation.cc:407] Connection header received
0928 08:31:21.676235 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:21.676239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:21.676306 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:21.676391 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:21.677370 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:21.678276 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:21.679527 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:21.679977 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:21.680096 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:21.684217 (+ 4121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.193:40577: BlockingRecv error: recv got EOF from 127.15.134.193:40577 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 08:31:22 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:22 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:22.153921 25134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:22.144889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.194:54331)
0928 08:31:22.145507 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:22.145525 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:22.145698 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:22.146224 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:22.146232 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:22.146709 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:22.147004 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:22.147015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:22.148019 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:22.148024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:22.149047 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:22.149060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:22.149849 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:22.149855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:22.149975 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:22.150718 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:22.150737 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:22.153670 (+ 2933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":364,"threads_started":1}
W20260928 08:31:22.154335 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:22.144996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:54331 (local address 127.15.134.254:39675)
0928 08:31:22.145213 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:22.145220 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:22.145595 (+ 375us) server_negotiation.cc:407] Connection header received
0928 08:31:22.145950 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:22.145956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:22.146044 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:22.146142 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:22.147145 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:22.147874 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:22.149187 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:22.149610 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:22.149704 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:22.154011 (+ 4307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.194:54331: BlockingRecv error: recv got EOF from 127.15.134.194:54331 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 08:31:22 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:22 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:22.632210 25135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:22.623489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.195:47737)
0928 08:31:22.624064 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:22.624079 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:22.624251 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:22.624639 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:22.624644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:22.625073 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:22.625241 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:22.625250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:22.626281 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:22.626287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:22.627267 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:22.627279 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:22.627990 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:22.627996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:22.628173 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:22.628899 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:22.628917 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:22.631866 (+ 2949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":278,"threads_started":1}
W20260928 08:31:22.632373 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:22.623690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:47737 (local address 127.15.134.254:39675)
0928 08:31:22.623916 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:22.623924 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:22.624167 (+ 243us) server_negotiation.cc:407] Connection header received
0928 08:31:22.624395 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:22.624399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:22.624474 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:22.624584 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:22.625411 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:22.626133 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:22.627413 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:22.627830 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:22.627973 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:22.632094 (+ 4121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.195:47737: BlockingRecv error: recv got EOF from 127.15.134.195:47737 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 28 08:31:22 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:22 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:22.694449 25136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:22.685280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.193:43331)
0928 08:31:22.685858 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:22.685876 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:22.686011 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:22.686329 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:22.686332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:22.686708 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:22.686903 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:22.686915 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:22.687931 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:22.687935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:22.688962 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:22.688974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:22.689722 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:22.689727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:22.689874 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:22.690558 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:22.690577 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:22.694087 (+ 3510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":317,"threads_started":1}
W20260928 08:31:22.694728 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:22.685394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:43331 (local address 127.15.134.254:39675)
0928 08:31:22.685562 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:22.685566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:22.685961 (+ 395us) server_negotiation.cc:407] Connection header received
0928 08:31:22.686119 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:22.686123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:22.686188 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:22.686277 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:22.687046 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:22.687777 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:22.689105 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:22.689574 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:22.689689 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:22.694418 (+ 4729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.193:43331: BlockingRecv error: recv got EOF from 127.15.134.193:43331 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 08:31:23 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:23 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:23.164244 25137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:23.155017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.194:43813)
0928 08:31:23.155594 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:23.155609 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:23.155902 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:23.156286 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:23.156293 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:23.156743 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:23.156957 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:23.156973 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:23.158038 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:23.158043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:23.158969 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:23.158981 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:23.159791 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:23.159797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:23.159982 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:23.160604 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:23.160681 (+ 77us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:23.163972 (+ 3291us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":327,"threads_started":1}
W20260928 08:31:23.164541 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:23.155141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:43813 (local address 127.15.134.254:39675)
0928 08:31:23.155446 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:23.155453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:23.155700 (+ 247us) server_negotiation.cc:407] Connection header received
0928 08:31:23.156042 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:23.156047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:23.156124 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:23.156243 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:23.157112 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:23.157889 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:23.159116 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:23.159620 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:23.159762 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:23.164242 (+ 4480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.194:43813: BlockingRecv error: recv got EOF from 127.15.134.194:43813 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
Sep 28 08:31:23 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:23 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:23.641852 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:23.633170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.195:52461)
0928 08:31:23.633685 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:23.633704 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:23.633856 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:23.634193 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:23.634196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:23.634582 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:23.634747 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:23.634757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:23.635696 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:23.635700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:23.636663 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:23.636676 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:23.637569 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:23.637576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:23.637794 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:23.638576 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:23.638601 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:23.641595 (+ 2994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":256,"threads_started":1}
W20260928 08:31:23.642077 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:23.633385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:52461 (local address 127.15.134.254:39675)
0928 08:31:23.633597 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:23.633605 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:23.633796 (+ 191us) server_negotiation.cc:407] Connection header received
0928 08:31:23.633971 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:23.633974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:23.634038 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:23.634121 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:23.634888 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:23.635582 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:23.636803 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:23.637397 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:23.637522 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:23.641847 (+ 4325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.195:52461: BlockingRecv error: recv got EOF from 127.15.134.195:52461 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 28 08:31:23 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:23 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:23.705479 25139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:23.695675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.193:54993)
0928 08:31:23.696314 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:23.696333 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:23.696490 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:23.696952 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:23.696956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:23.697328 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:23.697485 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:23.697494 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:23.698628 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:23.698634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:23.700049 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:23.700063 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:23.701027 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:23.701031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:23.701150 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:23.701839 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:23.701872 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:23.705157 (+ 3285us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":396,"threads_started":1}
W20260928 08:31:23.705648 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:23.695774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:54993 (local address 127.15.134.254:39675)
0928 08:31:23.696043 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:23.696048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:23.696473 (+ 425us) server_negotiation.cc:407] Connection header received
0928 08:31:23.696665 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:23.696671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:23.696755 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:23.696854 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:23.697628 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:23.698441 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:23.700252 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:23.700885 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:23.701009 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:23.705422 (+ 4413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.193:54993: BlockingRecv error: recv got EOF from 127.15.134.193:54993 (error 108)
Metrics: {"server-negotiator.queue_time_us":134}
Sep 28 08:31:24 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:24 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:24.175036 25140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:24.165358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.194:58701)
0928 08:31:24.165979 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:24.165997 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:24.166176 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:24.166580 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:24.166585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:24.167133 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:24.167299 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:24.167309 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:24.168389 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:24.168398 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:24.169540 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:24.169553 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:24.170335 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:24.170341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:24.170580 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:24.171350 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:24.171377 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:24.174704 (+ 3327us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":417,"threads_started":1}
W20260928 08:31:24.175151 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:24.165485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:58701 (local address 127.15.134.254:39675)
0928 08:31:24.165758 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:24.165767 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:24.166100 (+ 333us) server_negotiation.cc:407] Connection header received
0928 08:31:24.166336 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:24.166349 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:24.166418 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:24.166547 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:24.167465 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:24.168244 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:24.169700 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:24.170160 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:24.170326 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:24.174918 (+ 4592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.194:58701: BlockingRecv error: recv got EOF from 127.15.134.194:58701 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Sep 28 08:31:24 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:24 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:24.652505 25141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:24.642835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.195:49179)
0928 08:31:24.643351 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:24.643373 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:24.643486 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:24.643802 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:24.643806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:24.644200 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:24.644383 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:24.644392 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:24.645626 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:24.645630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:24.646609 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:24.646623 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:24.647348 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:24.647352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:24.647482 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:24.648091 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:24.648141 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:24.652192 (+ 4051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":282,"threads_started":1}
W20260928 08:31:24.652830 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:24.642948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:49179 (local address 127.15.134.254:39675)
0928 08:31:24.643150 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:24.643155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:24.643458 (+ 303us) server_negotiation.cc:407] Connection header received
0928 08:31:24.643587 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:24.643591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:24.643662 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:24.643756 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:24.644540 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:24.645496 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:24.646764 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:24.647194 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:24.647300 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:24.652520 (+ 5220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.195:49179: BlockingRecv error: recv got EOF from 127.15.134.195:49179 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 08:31:24 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:24 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:24.715683 25142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:24.706607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.193:41989)
0928 08:31:24.707136 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:24.707152 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:24.707316 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:24.707684 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:24.707689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:24.708185 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:24.708351 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:24.708360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:24.709442 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:24.709449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:24.710391 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:24.710403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:24.711132 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:24.711139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:24.711328 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:24.712061 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:24.712081 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:24.715299 (+ 3218us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":305,"threads_started":1}
W20260928 08:31:24.715760 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:24.706679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:41989 (local address 127.15.134.254:39675)
0928 08:31:24.706900 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:24.706909 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:24.707244 (+ 335us) server_negotiation.cc:407] Connection header received
0928 08:31:24.707473 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:24.707483 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:24.707547 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:24.707652 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:24.708525 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:24.709293 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:24.710541 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:24.710969 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:24.711103 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:24.715543 (+ 4440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.193:41989: BlockingRecv error: recv got EOF from 127.15.134.193:41989 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 08:31:25 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:25 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:25.184743 25143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:25.176171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.194:48985)
0928 08:31:25.176783 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:25.176841 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:25.176979 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:25.177400 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:25.177408 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:25.177809 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:25.177971 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:25.177980 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:25.178977 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:25.178980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:25.179881 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:25.179893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:25.180561 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:25.180565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:25.180718 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:25.181415 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:25.181433 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:25.184468 (+ 3035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":311,"threads_started":1}
W20260928 08:31:25.185108 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:25.176353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:48985 (local address 127.15.134.254:39675)
0928 08:31:25.176557 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:25.176561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:25.176926 (+ 365us) server_negotiation.cc:407] Connection header received
0928 08:31:25.177097 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:25.177101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:25.177219 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:25.177330 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:25.178098 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:25.178858 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:25.180013 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:25.180428 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:25.180523 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:25.184785 (+ 4262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.194:48985: BlockingRecv error: recv got EOF from 127.15.134.194:48985 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 08:31:25 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:25 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:25.664940 25144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:25.653910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.195:46767)
0928 08:31:25.654670 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:25.654686 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:25.654898 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:25.655442 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:25.655450 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:25.656021 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:25.656248 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:25.656268 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:25.658150 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:25.658155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:25.659585 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:25.659598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:25.660439 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:25.660443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:25.660570 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:25.661274 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:25.661293 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:25.664597 (+ 3304us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":471,"threads_started":1}
W20260928 08:31:25.665216 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:25.653991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:46767 (local address 127.15.134.254:39675)
0928 08:31:25.654230 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:25.654239 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:25.654789 (+ 550us) server_negotiation.cc:407] Connection header received
0928 08:31:25.655072 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:25.655081 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:25.655197 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:25.655344 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:25.656412 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:25.657951 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:25.659817 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:25.660294 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:25.660407 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:25.664927 (+ 4520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.195:46767: BlockingRecv error: recv got EOF from 127.15.134.195:46767 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 08:31:25 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:25 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:25.726197 25145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:25.717172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.193:44867)
0928 08:31:25.717752 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:25.717769 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:25.717941 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:25.718440 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:25.718448 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:25.718918 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:25.719125 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:25.719135 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:25.720500 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:25.720504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:25.721566 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:25.721579 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:25.722440 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:25.722448 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:25.722587 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:25.723217 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:25.723236 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:25.725983 (+ 2747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":317,"threads_started":1}
W20260928 08:31:25.726533 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:25.717344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:44867 (local address 127.15.134.254:39675)
0928 08:31:25.717628 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:25.717634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:25.717861 (+ 227us) server_negotiation.cc:407] Connection header received
0928 08:31:25.718159 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:25.718171 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:25.718266 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:25.718380 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:25.719398 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:25.720378 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:25.721734 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:25.722263 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:25.722377 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:25.726219 (+ 3842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.193:44867: BlockingRecv error: recv got EOF from 127.15.134.193:44867 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 08:31:26 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:26 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:26.194490 25146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:26.185801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.194:57195)
0928 08:31:26.186371 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:26.186387 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:26.186499 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:26.186815 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:26.186819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:26.187224 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:26.187377 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:26.187386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:26.188399 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:26.188403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:26.189414 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:26.189426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:26.190118 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:26.190123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:26.190311 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:26.190988 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:26.191006 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:26.194269 (+ 3263us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":317,"threads_started":1}
W20260928 08:31:26.194721 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:26.185932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:57195 (local address 127.15.134.254:39675)
0928 08:31:26.186107 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:26.186111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:26.186474 (+ 363us) server_negotiation.cc:407] Connection header received
0928 08:31:26.186598 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:26.186602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:26.186663 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:26.186748 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:26.187514 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:26.188269 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:26.189536 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:26.189955 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:26.190094 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:26.194495 (+ 4401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.194:57195: BlockingRecv error: recv got EOF from 127.15.134.194:57195 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 08:31:26 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:26 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:26.679926 25147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:26.666060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.195:52929)
0928 08:31:26.666633 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:26.666727 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:26.666889 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:26.667251 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:26.667254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:26.667636 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:26.667879 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:26.667890 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:26.669094 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:26.669099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:26.670619 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:26.670667 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:26.671653 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:26.671657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:26.671788 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:26.676423 (+ 4635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:26.676443 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:26.679615 (+ 3172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":337,"threads_started":1}
W20260928 08:31:26.680135 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:26.666143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:52929 (local address 127.15.134.254:39675)
0928 08:31:26.666422 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:26.666430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:26.666819 (+ 389us) server_negotiation.cc:407] Connection header received
0928 08:31:26.667040 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:26.667045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:26.667113 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:26.667208 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:26.668034 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:26.668907 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:26.670801 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:26.671480 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:26.671600 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:26.679868 (+ 8268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.195:52929: BlockingRecv error: recv got EOF from 127.15.134.195:52929 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 28 08:31:26 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:26 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:26.736413 25148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:26.727226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.193:45127)
0928 08:31:26.727908 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:26.727926 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:26.728046 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:26.728441 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:26.728444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:26.728925 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:26.729099 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:26.729108 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:26.730209 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:26.730214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:26.731598 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:26.731614 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:26.732563 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:26.732568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:26.732712 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:26.733360 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:26.733377 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:26.736160 (+ 2783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":287,"threads_started":1}
W20260928 08:31:26.736707 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:26.727292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:45127 (local address 127.15.134.254:39675)
0928 08:31:26.727505 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:26.727510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:26.728015 (+ 505us) server_negotiation.cc:407] Connection header received
0928 08:31:26.728177 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:26.728184 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:26.728290 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:26.728387 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:26.729242 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:26.730049 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:26.731814 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:26.732391 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:26.732491 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:26.736381 (+ 3890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.193:45127: BlockingRecv error: recv got EOF from 127.15.134.193:45127 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 08:31:27 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:27 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:27.204970 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:27.195552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.194:39251)
0928 08:31:27.196140 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:27.196157 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:27.196284 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:27.196800 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:27.196804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:27.197189 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:27.197355 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:27.197365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:27.198460 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:27.198463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:27.199382 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:27.199393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:27.200269 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:27.200274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:27.200400 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:27.201081 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:27.201101 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:27.204495 (+ 3394us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":283,"threads_started":1}
W20260928 08:31:27.205060 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:27.195621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:39251 (local address 127.15.134.254:39675)
0928 08:31:27.195993 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:27.196001 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:27.196247 (+ 246us) server_negotiation.cc:407] Connection header received
0928 08:31:27.196455 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:27.196465 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:27.196541 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:27.196722 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:27.197519 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:27.198324 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:27.199536 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:27.200079 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:27.200208 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:27.204757 (+ 4549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.194:39251: BlockingRecv error: recv got EOF from 127.15.134.194:39251 (error 108)
Metrics: {"server-negotiator.queue_time_us":168}
Sep 28 08:31:27 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:27 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:27.690044 25150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:27.680997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.195:55499)
0928 08:31:27.681549 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:27.681566 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:27.681725 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:27.682123 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:27.682128 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:27.682549 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:27.682715 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:27.682724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:27.683797 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:27.683803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:27.684827 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:27.684839 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:27.685581 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:27.685586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:27.685764 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:27.686336 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:27.686353 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:27.689670 (+ 3317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":235,"threads_started":1}
W20260928 08:31:27.690111 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:27.681024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:55499 (local address 127.15.134.254:39675)
0928 08:31:27.681189 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:27.681194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:27.681653 (+ 459us) server_negotiation.cc:407] Connection header received
0928 08:31:27.681912 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:27.681916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:27.681980 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:27.682079 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:27.682869 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:27.683643 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:27.684969 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:27.685424 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:27.685566 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:27.689876 (+ 4310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.195:55499: BlockingRecv error: recv got EOF from 127.15.134.195:55499 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 08:31:27 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:27 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:27.746716 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:27.737512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.193:44217)
0928 08:31:27.738327 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:27.738344 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:27.738520 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:27.738885 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:27.738890 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:27.739308 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:27.739456 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:27.739465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:27.740700 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:27.740706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:27.741684 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:27.741697 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:27.742498 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:27.742502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:27.742625 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:27.743385 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:27.743404 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:27.746468 (+ 3064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":721,"threads_started":1}
W20260928 08:31:27.747213 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:27.737883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:44217 (local address 127.15.134.254:39675)
0928 08:31:27.738121 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:27.738128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:27.738444 (+ 316us) server_negotiation.cc:407] Connection header received
0928 08:31:27.738653 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:27.738659 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:27.738739 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:27.738852 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:27.739630 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:27.740463 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:27.741891 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:27.742336 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:27.742434 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:27.746716 (+ 4282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.193:44217: BlockingRecv error: recv got EOF from 127.15.134.193:44217 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 28 08:31:28 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:28 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:28.215298 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:28.206188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.194:39129)
0928 08:31:28.206831 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:28.206848 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:28.207009 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:28.207355 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:28.207359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:28.207776 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:28.207919 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:28.207928 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:28.209569 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:28.209577 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:28.210684 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:28.210695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:28.211418 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:28.211424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:28.211581 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:28.212228 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:28.212245 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:28.215061 (+ 2816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":302,"threads_started":1}
W20260928 08:31:28.215484 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:28.206268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:39129 (local address 127.15.134.254:39675)
0928 08:31:28.206492 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:28.206501 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:28.206943 (+ 442us) server_negotiation.cc:407] Connection header received
0928 08:31:28.207136 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:28.207156 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:28.207218 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:28.207323 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:28.208067 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:28.209419 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:28.210819 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:28.211231 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:28.211363 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:28.215264 (+ 3901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.194:39129: BlockingRecv error: recv got EOF from 127.15.134.194:39129 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 08:31:28 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:28 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:28.700130 25153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:28.691216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.195:59683)
0928 08:31:28.692061 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:28.692082 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:28.692233 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:28.692861 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:28.692864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:28.693239 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:28.693387 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:28.693396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:28.694415 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:28.694422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:28.695334 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:28.695345 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:28.696061 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:28.696068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:28.696231 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:28.696979 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:28.696997 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:28.699790 (+ 2793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":547,"threads_started":1}
W20260928 08:31:28.700268 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:28.691319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:59683 (local address 127.15.134.254:39675)
0928 08:31:28.691527 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:28.691533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:28.692235 (+ 702us) server_negotiation.cc:407] Connection header received
0928 08:31:28.692405 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:28.692426 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:28.692539 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:28.692660 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:28.693525 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:28.694264 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:28.695471 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:28.695883 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:28.696029 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:28.699991 (+ 3962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.195:59683: BlockingRecv error: recv got EOF from 127.15.134.195:59683 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 08:31:28 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:28 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:28.756825 25154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:28.747794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.193:50025)
0928 08:31:28.748320 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:28.748337 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:28.748462 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:28.748803 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:28.748806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:28.749207 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:28.749386 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:28.749395 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:28.750447 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:28.750451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:28.751463 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:28.751477 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:28.752214 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:28.752218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:28.752363 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:28.753009 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:28.753032 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:28.756436 (+ 3404us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":285,"threads_started":1}
W20260928 08:31:28.756966 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:28.747914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:50025 (local address 127.15.134.254:39675)
0928 08:31:28.748212 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:28.748218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:28.748427 (+ 209us) server_negotiation.cc:407] Connection header received
0928 08:31:28.748568 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:28.748572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:28.748661 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:28.748747 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:28.749532 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:28.750290 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:28.751616 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:28.752057 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:28.752177 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:28.756678 (+ 4501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.193:50025: BlockingRecv error: recv got EOF from 127.15.134.193:50025 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Sep 28 08:31:29 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:29 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:29.225414 25155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:29.216265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.194:42019)
0928 08:31:29.217096 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:29.217112 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:29.217286 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:29.217669 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:29.217674 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:29.218097 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:29.218265 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:29.218275 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:29.219386 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:29.219393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:29.220434 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:29.220448 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:29.221375 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:29.221381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:29.221562 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:29.222215 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:29.222235 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:29.225154 (+ 2919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":606,"threads_started":1}
W20260928 08:31:29.225725 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:29.216493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:42019 (local address 127.15.134.254:39675)
0928 08:31:29.216703 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:29.216709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:29.217221 (+ 512us) server_negotiation.cc:407] Connection header received
0928 08:31:29.217435 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:29.217440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:29.217512 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:29.217621 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:29.218422 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:29.219241 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:29.220590 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:29.221197 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:29.221348 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:29.225417 (+ 4069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.194:42019: BlockingRecv error: recv got EOF from 127.15.134.194:42019 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 08:31:29 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:29 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:29.711107 25156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:29.701366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.195:49643)
0928 08:31:29.702004 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:29.702023 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:29.702159 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:29.702645 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:29.702651 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:29.703105 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:29.703333 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:29.703349 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:29.704495 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:29.704502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:29.705611 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:29.705627 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:29.706527 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:29.706534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:29.706746 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:29.707445 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:29.707466 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:29.710846 (+ 3380us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":357,"threads_started":1}
W20260928 08:31:29.711540 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:29.701472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:49643 (local address 127.15.134.254:39675)
0928 08:31:29.701693 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:29.701700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:29.702121 (+ 421us) server_negotiation.cc:407] Connection header received
0928 08:31:29.702311 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:29.702318 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:29.702403 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:29.702540 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:29.703510 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:29.704334 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:29.705790 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:29.706333 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:29.706494 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:29.711095 (+ 4601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.195:49643: BlockingRecv error: recv got EOF from 127.15.134.195:49643 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 28 08:31:29 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:29 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:29.766601 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:29.757909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.193:42027)
0928 08:31:29.758411 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:29.758428 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:29.758539 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:29.758940 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:29.758944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:29.759323 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:29.759475 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:29.759483 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:29.760721 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:29.760725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:29.761697 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:29.761709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:29.762513 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:29.762517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:29.762633 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:29.763305 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:29.763322 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:29.766368 (+ 3046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":278,"threads_started":1}
W20260928 08:31:29.766912 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:29.758014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:42027 (local address 127.15.134.254:39675)
0928 08:31:29.758311 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:29.758318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:29.758512 (+ 194us) server_negotiation.cc:407] Connection header received
0928 08:31:29.758721 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:29.758725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:29.758798 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:29.758895 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:29.759700 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:29.760522 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:29.761886 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:29.762357 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:29.762474 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:29.766570 (+ 4096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.193:42027: BlockingRecv error: recv got EOF from 127.15.134.193:42027 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Sep 28 08:31:30 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:30 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:30.234752 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:30.226321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.194:42543)
0928 08:31:30.226841 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:30.226856 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:30.226965 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:30.227299 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:30.227302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:30.227662 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:30.227933 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:30.227943 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:30.228976 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:30.228980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:30.229900 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:30.229912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:30.230553 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:30.230557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:30.230667 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:30.231319 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:30.231337 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:30.234456 (+ 3119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":289,"threads_started":1}
W20260928 08:31:30.235066 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:30.226364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:42543 (local address 127.15.134.254:39675)
0928 08:31:30.226544 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:30.226548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:30.226939 (+ 391us) server_negotiation.cc:407] Connection header received
0928 08:31:30.227097 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:30.227101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:30.227170 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:30.227263 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:30.228099 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:30.228868 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:30.230024 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:30.230429 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:30.230524 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:30.234777 (+ 4253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.194:42543: BlockingRecv error: recv got EOF from 127.15.134.194:42543 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 08:31:30 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:30 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:30.720824 25159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:30.712215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.195:60771)
0928 08:31:30.712811 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:30.712828 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:30.712970 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:30.713316 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:30.713321 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:30.713740 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:30.713879 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:30.713888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:30.714905 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:30.714910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:30.715967 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:30.715979 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:30.716773 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:30.716779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:30.716948 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:30.717520 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:30.717537 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:30.720474 (+ 2937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":310,"threads_started":1}
W20260928 08:31:30.721072 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:30.712437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:60771 (local address 127.15.134.254:39675)
0928 08:31:30.712694 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:30.712701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:30.712904 (+ 203us) server_negotiation.cc:407] Connection header received
0928 08:31:30.713098 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:30.713103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:30.713180 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:30.713293 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:30.714018 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:30.714766 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:30.716104 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:30.716570 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:30.716756 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:30.720787 (+ 4031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.195:60771: BlockingRecv error: recv got EOF from 127.15.134.195:60771 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 08:31:30 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:30 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:30.776939 25160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:30.767553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.193:46565)
0928 08:31:30.768198 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:30.768216 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:30.768340 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:30.768670 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:30.768674 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:30.769165 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:30.769354 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:30.769365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:30.770394 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:30.770398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:30.771429 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:30.771447 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:30.772378 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:30.772386 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:30.772560 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:30.773512 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:30.773539 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:30.776672 (+ 3133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":389,"threads_started":1}
W20260928 08:31:30.777112 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:30.767727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:46565 (local address 127.15.134.254:39675)
0928 08:31:30.767923 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:30.767928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:30.768324 (+ 396us) server_negotiation.cc:407] Connection header received
0928 08:31:30.768444 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:30.768448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:30.768515 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:30.768602 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:30.769504 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:30.770260 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:30.771603 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:30.772217 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:30.772318 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:30.776878 (+ 4560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.193:46565: BlockingRecv error: recv got EOF from 127.15.134.193:46565 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 08:31:31 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:31 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:31.244400 25161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:31.235642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.194:40489)
0928 08:31:31.236130 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:31.236145 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:31.236296 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:31.236690 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:31.236695 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:31.237112 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:31.237283 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:31.237292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:31.238300 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:31.238317 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:31.239203 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:31.239214 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:31.239947 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:31.239976 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:31.240162 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:31.240938 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:31.240980 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:31.244144 (+ 3164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":259,"threads_started":1}
W20260928 08:31:31.244642 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:31.235734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:40489 (local address 127.15.134.254:39675)
0928 08:31:31.235975 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:31.235982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:31.236223 (+ 241us) server_negotiation.cc:407] Connection header received
0928 08:31:31.236456 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:31.236465 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:31.236535 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:31.236658 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:31.237433 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:31.238165 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:31.239360 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:31.239776 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:31.239922 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:31.244368 (+ 4446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.194:40489: BlockingRecv error: recv got EOF from 127.15.134.194:40489 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 28 08:31:31 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:31 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:31.736384 25162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:31.722017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.195:51827)
0928 08:31:31.722715 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:31.722748 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:31.722906 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:31.723304 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:31.723309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:31.723976 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:31.724252 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:31.724268 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:31.725514 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:31.725519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:31.726445 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:31.726457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:31.727285 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:31.727290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:31.727424 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:31.732870 (+ 5446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:31.732895 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:31.736052 (+ 3157us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":324,"threads_started":1}
W20260928 08:31:31.736683 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:31.722128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:51827 (local address 127.15.134.254:39675)
0928 08:31:31.722388 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:31.722396 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:31.722875 (+ 479us) server_negotiation.cc:407] Connection header received
0928 08:31:31.723039 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:31.723053 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:31.723139 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:31.723223 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:31.724456 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:31.725380 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:31.726589 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:31.727124 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:31.727226 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:31.736383 (+ 9157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.195:51827: BlockingRecv error: recv got EOF from 127.15.134.195:51827 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 08:31:31 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:31 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:31.787474 25163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:31.778053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.193:47313)
0928 08:31:31.778535 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:31.778552 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:31.778707 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:31.779062 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:31.779066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:31.779513 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:31.779696 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:31.779706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:31.780886 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:31.780893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:31.781927 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:31.781941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:31.782773 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:31.782778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:31.782917 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:31.783591 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:31.783684 (+ 93us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:31.787245 (+ 3561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":273,"threads_started":1}
W20260928 08:31:31.787660 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:31.778137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:47313 (local address 127.15.134.254:39675)
0928 08:31:31.778410 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:31.778417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:31.778646 (+ 229us) server_negotiation.cc:407] Connection header received
0928 08:31:31.778829 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:31.778833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:31.778897 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:31.778984 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:31.779867 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:31.780658 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:31.782063 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:31.782632 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:31.782735 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:31.787412 (+ 4677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.193:47313: BlockingRecv error: recv got EOF from 127.15.134.193:47313 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 08:31:32 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:32 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:32.253463 25164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:32.245326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.194:36977)
0928 08:31:32.245780 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:32.245796 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:32.245920 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:32.246204 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:32.246207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:32.246555 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:32.246701 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:32.246708 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:32.247674 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:32.247677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:32.248657 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:32.248669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:32.249301 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:32.249306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:32.249433 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:32.250005 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:32.250025 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:32.253232 (+ 3207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":265,"threads_started":1}
W20260928 08:31:32.253799 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:32.245427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:36977 (local address 127.15.134.254:39675)
0928 08:31:32.245694 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:32.245700 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:32.245876 (+ 176us) server_negotiation.cc:407] Connection header received
0928 08:31:32.246020 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:32.246023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:32.246087 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:32.246188 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:32.246820 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:32.247544 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:32.248771 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:32.249158 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:32.249270 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:32.253428 (+ 4158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.194:36977: BlockingRecv error: recv got EOF from 127.15.134.194:36977 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 28 08:31:32 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:32 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:32.746554 25165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:32.737523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.195:40837)
0928 08:31:32.738013 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:32.738031 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:32.738175 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:32.738481 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:32.738484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:32.738964 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:32.739137 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:32.739147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:32.740170 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:32.740180 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:32.741206 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:32.741219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:32.741959 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:32.741964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:32.742159 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:32.742750 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:32.742767 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:32.746162 (+ 3395us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":273,"threads_started":1}
W20260928 08:31:32.746726 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:32.737649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:40837 (local address 127.15.134.254:39675)
0928 08:31:32.737891 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:32.737898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:32.738115 (+ 217us) server_negotiation.cc:407] Connection header received
0928 08:31:32.738285 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:32.738289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:32.738355 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:32.738437 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:32.739269 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:32.740007 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:32.741351 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:32.741814 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:32.741946 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:32.746382 (+ 4436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.195:40837: BlockingRecv error: recv got EOF from 127.15.134.195:40837 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 28 08:31:32 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:32 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:32.797824 25166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:32.788483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.193:33217)
0928 08:31:32.789082 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:32.789098 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:32.789214 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:32.789524 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:32.789527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:32.789996 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:32.790155 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:32.790164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:32.791380 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:32.791394 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:32.792495 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:32.792508 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:32.793426 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:32.793431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:32.793549 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:32.794283 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:32.794310 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:32.797546 (+ 3236us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":341,"threads_started":1}
W20260928 08:31:32.798022 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:32.788760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:33217 (local address 127.15.134.254:39675)
0928 08:31:32.788951 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:32.788955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:32.789175 (+ 220us) server_negotiation.cc:407] Connection header received
0928 08:31:32.789315 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:32.789319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:32.789393 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:32.789487 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:32.790294 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:32.791184 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:32.792676 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:32.793243 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:32.793342 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:32.797780 (+ 4438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.193:33217: BlockingRecv error: recv got EOF from 127.15.134.193:33217 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 08:31:33 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:33 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:33.263409 25167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:33.254391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.194:53583)
0928 08:31:33.254931 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:33.254947 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:33.255062 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:33.255371 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:33.255374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:33.255766 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:33.255918 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:33.255926 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:33.257128 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:33.257132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:33.258077 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:33.258089 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:33.258824 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:33.258828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:33.258940 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:33.259675 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:33.259701 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:33.263118 (+ 3417us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":286,"threads_started":1}
W20260928 08:31:33.263514 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:33.254605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:53583 (local address 127.15.134.254:39675)
0928 08:31:33.254783 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:33.254788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:33.255042 (+ 254us) server_negotiation.cc:407] Connection header received
0928 08:31:33.255173 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:33.255177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:33.255250 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:33.255335 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:33.256078 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:33.256995 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:33.258212 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:33.258667 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:33.258776 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:33.263318 (+ 4542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.194:53583: BlockingRecv error: recv got EOF from 127.15.134.194:53583 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 08:31:33 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:33 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:33.757027 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:33.747855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.195:46993)
0928 08:31:33.748472 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:33.748490 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:33.748658 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:33.748987 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:33.748990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:33.749417 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:33.749684 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:33.749696 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:33.750769 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:33.750773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:33.751867 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:33.751888 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:33.752793 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:33.752801 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:33.752973 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:33.753711 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:33.753730 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:33.756780 (+ 3050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":339,"threads_started":1}
W20260928 08:31:33.757313 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:33.747902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:46993 (local address 127.15.134.254:39675)
0928 08:31:33.748094 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:33.748099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:33.748582 (+ 483us) server_negotiation.cc:407] Connection header received
0928 08:31:33.748776 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:33.748780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:33.748848 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:33.748937 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:33.749858 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:33.750608 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:33.752044 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:33.752468 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:33.752571 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:33.756990 (+ 4419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.195:46993: BlockingRecv error: recv got EOF from 127.15.134.195:46993 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 28 08:31:33 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:33 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:33.807199 25169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:33.798783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.193:57163)
0928 08:31:33.799234 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:33.799252 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:33.799366 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:33.799659 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:33.799664 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:33.800057 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:33.800256 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:33.800265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:33.801354 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:33.801358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:33.802273 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:33.802285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:33.802976 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:33.802981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:33.803169 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:33.803839 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:33.803857 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:33.806963 (+ 3106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":241,"threads_started":1}
W20260928 08:31:33.807412 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:33.798891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:57163 (local address 127.15.134.254:39675)
0928 08:31:33.799215 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:33.799221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:33.799341 (+ 120us) server_negotiation.cc:407] Connection header received
0928 08:31:33.799464 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:33.799468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:33.799533 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:33.799614 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:33.800389 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:33.801234 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:33.802394 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:33.802792 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:33.802881 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:33.807153 (+ 4272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.193:57163: BlockingRecv error: recv got EOF from 127.15.134.193:57163 (error 108)
Metrics: {"server-negotiator.queue_time_us":166}
Sep 28 08:31:34 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:34 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:34.273345 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:34.264565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.194:41323)
0928 08:31:34.265099 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:34.265115 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:34.265228 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:34.265531 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:34.265535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:34.265912 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:34.266127 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:34.266137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:34.267164 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:34.267168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:34.268147 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:34.268160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:34.268873 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:34.268877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:34.269007 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:34.269614 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:34.269638 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:34.272962 (+ 3324us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":262,"threads_started":1}
W20260928 08:31:34.273376 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:34.264680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:41323 (local address 127.15.134.254:39675)
0928 08:31:34.264859 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:34.264863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:34.265188 (+ 325us) server_negotiation.cc:407] Connection header received
0928 08:31:34.265339 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:34.265342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:34.265407 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:34.265483 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:34.266269 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:34.266996 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:34.268278 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:34.268706 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:34.268794 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:34.273168 (+ 4374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.194:41323: BlockingRecv error: recv got EOF from 127.15.134.194:41323 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 08:31:34 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:34 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.195@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:34.766357 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:34.757904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.195:36815)
0928 08:31:34.758392 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:34.758410 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:34.758547 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:34.758881 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:34.758885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:34.759269 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:34.759462 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:34.759471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:34.760733 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:34.760737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:34.761673 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:34.761685 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:34.762366 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:34.762370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:34.762483 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:34.763183 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:34.763201 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:34.766026 (+ 2825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":262,"threads_started":1}
W20260928 08:31:34.766530 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:34.758130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:36815 (local address 127.15.134.254:39675)
0928 08:31:34.758350 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:34.758356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:34.758488 (+ 132us) server_negotiation.cc:407] Connection header received
0928 08:31:34.758676 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:34.758680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:34.758745 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:34.758833 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:34.759716 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:34.760449 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:34.761813 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:34.762247 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:34.762345 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:34.766290 (+ 3945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.195:36815: BlockingRecv error: recv got EOF from 127.15.134.195:36815 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 28 08:31:34 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:34 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.193@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:34.816995 25172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:34.808326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.193:52693)
0928 08:31:34.808943 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:34.808959 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:34.809102 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:34.809424 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:34.809428 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:34.809836 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:34.810005 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:34.810014 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:34.811067 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:34.811072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:34.812010 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:34.812021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:34.812781 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:34.812787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:34.812967 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:34.813603 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:34.813635 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:34.816743 (+ 3108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":361,"threads_started":1}
W20260928 08:31:34.817328 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:34.808435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:52693 (local address 127.15.134.254:39675)
0928 08:31:34.808717 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:34.808725 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:34.809037 (+ 312us) server_negotiation.cc:407] Connection header received
0928 08:31:34.809228 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:34.809238 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:34.809297 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:34.809390 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:34.810210 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:34.810924 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:34.812163 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:34.812592 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:34.812761 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:34.816949 (+ 4188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.193:52693: BlockingRecv error: recv got EOF from 127.15.134.193:52693 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 08:31:35 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:31:35 dist-test-slave-qp2n krb5kdc[24127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.15.134.194@KRBTEST.COM for oryx/127.15.134.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:31:35.284231 25173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:35.274807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:39675 (local address 127.15.134.194:45339)
0928 08:31:35.275367 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:35.275383 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:35.275498 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:35.275907 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:35.275911 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:35.276339 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:35.276501 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:35.276510 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:35.277893 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:35.277899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:35.279197 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:35.279215 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:35.280035 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:35.280042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:35.280235 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:35.280944 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:35.280962 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:35.283928 (+ 2966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.15.134.254:39675: Server oryx/127.15.134.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":319,"threads_started":1}
W20260928 08:31:35.284369 24864 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:35.274964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:45339 (local address 127.15.134.254:39675)
0928 08:31:35.275137 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:35.275141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:35.275458 (+ 317us) server_negotiation.cc:407] Connection header received
0928 08:31:35.275599 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:35.275603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:35.275712 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:35.275807 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:35.276754 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:35.277726 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:35.279361 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:35.279865 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:35.280049 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:35.284140 (+ 4091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.15.134.194:45339: BlockingRecv error: recv got EOF from 127.15.134.194:45339 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 08:31:35.585690 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 24715
I20260928 08:31:35.598773 25175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:35.586923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40624 (local address 127.15.134.193:35663)
0928 08:31:35.587664 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:35.587672 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:35.587699 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:31:35.587762 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:35.587769 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:35.587992 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:35.588163 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:35.589636 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:35.590796 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:35.592183 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:35.592868 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:35.592970 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:35.595544 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:35.595553 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:35.595568 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:35.595610 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:35.598270 (+ 2660us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.15.134.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 08:31:35.598542 (+ 272us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40624: Request ticket server kudu/127.15.134.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":424,"threads_started":1}
W20260928 08:31:35.598899 25175 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40624: Request ticket server kudu/127.15.134.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 08:31:35.599020 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 24715: Not authorized: Client connection negotiation failed: client connection to 127.15.134.193:35663: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.15.134.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 08:31:35.599290 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 24715
I20260928 08:31:35.617566 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 24866
I20260928 08:31:35.631330 25176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:35.618764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41738 (local address 127.15.134.194:38821)
0928 08:31:35.619391 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:35.619399 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:35.619431 (+ 32us) server_negotiation.cc:407] Connection header received
0928 08:31:35.619491 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:35.619498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:35.619796 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:35.619957 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:35.621404 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:35.622529 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:35.623808 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:35.624490 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:35.624593 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:35.627772 (+ 3179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:35.627802 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:35.627824 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:35.627883 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:35.630804 (+ 2921us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.15.134.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 08:31:35.631094 (+ 290us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41738: Request ticket server kudu/127.15.134.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":296,"threads_started":1}
W20260928 08:31:35.631466 25176 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41738: Request ticket server kudu/127.15.134.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 08:31:35.631587 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 24866: Not authorized: Client connection negotiation failed: client connection to 127.15.134.194:38821: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.15.134.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 08:31:35.631747 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 24866
I20260928 08:31:35.649333 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 24997
I20260928 08:31:35.661617 25177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:35.650512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43946 (local address 127.15.134.195:42727)
0928 08:31:35.651123 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:35.651132 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:35.651158 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:31:35.651224 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:35.651230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:35.651452 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:35.651666 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:35.652933 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:35.653987 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:35.655188 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:35.655724 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:35.655824 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:35.658375 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:35.658383 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:35.658398 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:35.658435 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:35.661081 (+ 2646us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.15.134.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 08:31:35.661358 (+ 277us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43946: Request ticket server kudu/127.15.134.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":324,"threads_started":1}
W20260928 08:31:35.661729 25177 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43946: Request ticket server kudu/127.15.134.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 08:31:35.661789 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 24997: Not authorized: Client connection negotiation failed: client connection to 127.15.134.195:42727: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.15.134.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 08:31:35.661969 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 24997
I20260928 08:31:35.679958 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 24646
I20260928 08:31:35.681433 24708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:35.804050 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 24646
2026-09-28T08:31:35Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21330 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T08:31:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:31:35Z Disabled control of system clock
I20260928 08:31:35.886847 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:42463
--webserver_interface=127.15.134.254
--webserver_port=0
--builtin_ntp_servers=127.15.134.212:44761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:42463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 08:31:36.232930 25182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:36.233232 25182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:36.233311 25182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:36.240528 25182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:31:36.240653 25182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:36.240720 25182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:36.240806 25182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:31:36.240865 25182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:31:36.250245 25182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:44761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:42463
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:42463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:36.251633 25182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:36.253423 25182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:36.265507 25187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:36.265412 25188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:36.265440 25190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:36.266356 25182 server_base.cc:1034] running on GCE node
I20260928 08:31:36.267088 25182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:36.268497 25182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:36.269722 25182 hybrid_clock.cc:648] HybridClock initialized: now 1790584296269695 us; error 78 us; skew 500 ppm
I20260928 08:31:36.270054 25182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:31:36.272262 25182 webserver.cc:460] Webserver started at http://127.15.134.254:37491/ using document root <none> and password file <none>
I20260928 08:31:36.272998 25182 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:36.273118 25182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:36.273437 25182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:31:36.276262 25182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "bbf89fadc0ae45baab43ce11e9998af2"
format_stamp: "Formatted at 2026-09-28 08:31:36 on dist-test-slave-qp2n"
server_key: "8f730d370c5a7281505b4a2a6a38e410"
server_key_iv: "032d944295b844e80a6d73f36f81b09a"
server_key_version: "encryptionkey@0"
I20260928 08:31:36.277081 25182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "bbf89fadc0ae45baab43ce11e9998af2"
format_stamp: "Formatted at 2026-09-28 08:31:36 on dist-test-slave-qp2n"
server_key: "8f730d370c5a7281505b4a2a6a38e410"
server_key_iv: "032d944295b844e80a6d73f36f81b09a"
server_key_version: "encryptionkey@0"
I20260928 08:31:36.282294 25182 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:31:36.285799 25196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:36.287057 25182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:31:36.287230 25182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "bbf89fadc0ae45baab43ce11e9998af2"
format_stamp: "Formatted at 2026-09-28 08:31:36 on dist-test-slave-qp2n"
server_key: "8f730d370c5a7281505b4a2a6a38e410"
server_key_iv: "032d944295b844e80a6d73f36f81b09a"
server_key_version: "encryptionkey@0"
I20260928 08:31:36.287441 25182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:36.303889 25182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:36.304781 25182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:36.305102 25182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:36.324321 25182 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:42463
I20260928 08:31:36.324344 25247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:42463 every 8 connection(s)
I20260928 08:31:36.325935 25182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:31:36.329728 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 25182
I20260928 08:31:36.329916 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:31:36.330019 25248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:36.330650 15899 external_mini_cluster.cc:1428] Setting key a559271d267058ab7a7160004012ce3a
I20260928 08:31:36.345119 25248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbf89fadc0ae45baab43ce11e9998af2: Bootstrap starting.
I20260928 08:31:36.347877 25251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:36.332749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44842 (local address 127.15.134.254:42463)
0928 08:31:36.333909 (+ 1160us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:36.333925 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:36.333989 (+ 64us) server_negotiation.cc:407] Connection header received
0928 08:31:36.336842 (+ 2853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:36.336876 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:36.337576 (+ 700us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:36.338048 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:36.338867 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:36.340820 (+ 1953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:36.342228 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:36.343329 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:36.343547 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:36.344809 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:36.344821 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:36.344842 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:31:36.345622 (+ 780us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:36.345731 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:31:36.345760 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:36.345876 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:36.346123 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 08:31:36.346486 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":353,"threads_started":1}
I20260928 08:31:36.349162 25248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbf89fadc0ae45baab43ce11e9998af2: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:36.350274 25248 log.cc:824] T 00000000000000000000000000000000 P bbf89fadc0ae45baab43ce11e9998af2: Log is configured to *not* fsync() on all Append() calls
I20260928 08:31:36.353363 25248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbf89fadc0ae45baab43ce11e9998af2: No bootstrap required, opened a new log
I20260928 08:31:36.361809 25248 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbf89fadc0ae45baab43ce11e9998af2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf89fadc0ae45baab43ce11e9998af2" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 42463 } }
I20260928 08:31:36.362304 25248 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbf89fadc0ae45baab43ce11e9998af2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:36.362427 25248 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbf89fadc0ae45baab43ce11e9998af2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf89fadc0ae45baab43ce11e9998af2, State: Initialized, Role: FOLLOWER
I20260928 08:31:36.363108 25248 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbf89fadc0ae45baab43ce11e9998af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf89fadc0ae45baab43ce11e9998af2" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 42463 } }
I20260928 08:31:36.363416 25248 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbf89fadc0ae45baab43ce11e9998af2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:31:36.363590 25248 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbf89fadc0ae45baab43ce11e9998af2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:31:36.363775 25248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbf89fadc0ae45baab43ce11e9998af2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:36.366240 25248 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbf89fadc0ae45baab43ce11e9998af2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf89fadc0ae45baab43ce11e9998af2" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 42463 } }
I20260928 08:31:36.366813 25248 leader_election.cc:302] T 00000000000000000000000000000000 P bbf89fadc0ae45baab43ce11e9998af2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbf89fadc0ae45baab43ce11e9998af2; no voters:
I20260928 08:31:36.367405 25248 leader_election.cc:288] T 00000000000000000000000000000000 P bbf89fadc0ae45baab43ce11e9998af2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:31:36.367573 25253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbf89fadc0ae45baab43ce11e9998af2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:31:36.368347 25253 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbf89fadc0ae45baab43ce11e9998af2 [term 1 LEADER]: Becoming Leader. State: Replica: bbf89fadc0ae45baab43ce11e9998af2, State: Running, Role: LEADER
I20260928 08:31:36.368984 25253 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbf89fadc0ae45baab43ce11e9998af2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf89fadc0ae45baab43ce11e9998af2" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 42463 } }
I20260928 08:31:36.369787 25248 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbf89fadc0ae45baab43ce11e9998af2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:31:36.372145 25254 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbf89fadc0ae45baab43ce11e9998af2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbf89fadc0ae45baab43ce11e9998af2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf89fadc0ae45baab43ce11e9998af2" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 42463 } } }
I20260928 08:31:36.372757 25254 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbf89fadc0ae45baab43ce11e9998af2 [sys.catalog]: This master's current role is: LEADER
I20260928 08:31:36.372598 25255 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbf89fadc0ae45baab43ce11e9998af2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbf89fadc0ae45baab43ce11e9998af2. Latest consensus state: current_term: 1 leader_uuid: "bbf89fadc0ae45baab43ce11e9998af2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbf89fadc0ae45baab43ce11e9998af2" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 42463 } } }
I20260928 08:31:36.372958 25255 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbf89fadc0ae45baab43ce11e9998af2 [sys.catalog]: This master's current role is: LEADER
I20260928 08:31:36.373661 25260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:31:36.379529 25260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:31:36.389329 25260 catalog_manager.cc:1347] Generated new cluster ID: b3e10b495dbc44ebb2b1206e154f894a
I20260928 08:31:36.389467 25260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:31:36.425884 25260 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:31:36.427757 25260 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:31:36.437186 25260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbf89fadc0ae45baab43ce11e9998af2: Generated new TSK 0
I20260928 08:31:36.438227 25260 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:31:36.452126 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:0
--local_ip_for_outbound_sockets=127.15.134.193
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:42463
--builtin_ntp_servers=127.15.134.212:44761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 08:31:36.799741 25272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:36.800062 25272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:36.800205 25272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:36.807475 25272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:36.807616 25272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:36.807786 25272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:31:36.816959 25272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:44761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:42463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:36.818252 25272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:36.820022 25272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:36.832989 25277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:36.833050 25278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:36.833070 25280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:36.835104 25272 server_base.cc:1034] running on GCE node
I20260928 08:31:36.835834 25272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:36.836968 25272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:36.838176 25272 hybrid_clock.cc:648] HybridClock initialized: now 1790584296838123 us; error 43 us; skew 500 ppm
I20260928 08:31:36.838521 25272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:31:36.844110 25272 webserver.cc:460] Webserver started at http://127.15.134.193:46031/ using document root <none> and password file <none>
I20260928 08:31:36.845021 25272 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:36.845139 25272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:36.845468 25272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:31:36.848428 25272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/data/instance:
uuid: "daf06a2845c84d828d06bde80c995e62"
format_stamp: "Formatted at 2026-09-28 08:31:36 on dist-test-slave-qp2n"
server_key: "b06e77cfd49881c30984d463146982ff"
server_key_iv: "df2b745ff622022edc6ec18191d2c398"
server_key_version: "encryptionkey@0"
I20260928 08:31:36.849328 25272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance:
uuid: "daf06a2845c84d828d06bde80c995e62"
format_stamp: "Formatted at 2026-09-28 08:31:36 on dist-test-slave-qp2n"
server_key: "b06e77cfd49881c30984d463146982ff"
server_key_iv: "df2b745ff622022edc6ec18191d2c398"
server_key_version: "encryptionkey@0"
I20260928 08:31:36.854398 25272 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:31:36.857971 25286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:36.859570 25272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:31:36.859773 25272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "daf06a2845c84d828d06bde80c995e62"
format_stamp: "Formatted at 2026-09-28 08:31:36 on dist-test-slave-qp2n"
server_key: "b06e77cfd49881c30984d463146982ff"
server_key_iv: "df2b745ff622022edc6ec18191d2c398"
server_key_version: "encryptionkey@0"
I20260928 08:31:36.860008 25272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:36.878862 25272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:36.879748 25272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:36.880056 25272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:36.881126 25272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:31:36.882673 25272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:31:36.882809 25272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:36.882961 25272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:31:36.883054 25272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:36.917199 25272 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:35133
I20260928 08:31:36.917285 25398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:35133 every 8 connection(s)
I20260928 08:31:36.918659 25272 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:31:36.928402 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 25272
I20260928 08:31:36.928599 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance
I20260928 08:31:36.929359 15899 external_mini_cluster.cc:1428] Setting key 9a445de5feb2abe923aefe493e43a8d5
I20260928 08:31:36.931900 25401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:36.921346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:35985 (local address 127.15.134.254:42463)
0928 08:31:36.921770 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:36.921775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:36.923010 (+ 1235us) server_negotiation.cc:407] Connection header received
0928 08:31:36.924792 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:36.924798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:36.924892 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:36.925008 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:36.926074 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:36.926847 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:36.928993 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:36.929403 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:36.929575 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:36.931048 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:36.931057 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:36.931062 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:31:36.931083 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:36.931152 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:31:36.931179 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:36.931274 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:36.931554 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 08:31:36.931646 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":216,"threads_started":1}
I20260928 08:31:36.932441 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:0
--local_ip_for_outbound_sockets=127.15.134.194
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:42463
--builtin_ntp_servers=127.15.134.212:44761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 08:31:36.933379 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:36.921962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:42463 (local address 127.15.134.193:35985)
0928 08:31:36.922852 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:36.922896 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:36.924493 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:36.925240 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:36.925249 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:36.925334 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:36.925909 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:36.925926 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:36.927004 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:36.927009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:36.928806 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:36.928831 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:36.929668 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:36.929677 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:36.929955 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:36.930746 (+ 791us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:31:36.930748 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:36.930789 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:31:36.930866 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:36.931284 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:36.931292 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:36.931298 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 08:31:36.931512 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 08:31:36.931948 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":257,"threads_started":1}
I20260928 08:31:36.934929 25399 heartbeater.cc:347] Connected to a master server at 127.15.134.254:42463
I20260928 08:31:36.935256 25399 heartbeater.cc:464] Registering TS with master...
I20260928 08:31:36.936193 25399 heartbeater.cc:511] Master 127.15.134.254:42463 requested a full tablet report, sending...
I20260928 08:31:36.938519 25213 ts_manager.cc:194] Registered new tserver with Master: daf06a2845c84d828d06bde80c995e62 (127.15.134.193:35133)
I20260928 08:31:36.940150 25213 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.134.193:35985
W20260928 08:31:37.280575 25403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:37.280897 25403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:37.280990 25403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:37.289394 25403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:37.289525 25403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:37.289644 25403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:31:37.299672 25403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:44761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:42463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:37.300976 25403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:37.302702 25403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:37.316112 25411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:37.316247 25409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:37.316563 25408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:37.317487 25403 server_base.cc:1034] running on GCE node
I20260928 08:31:37.318130 25403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:37.319183 25403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:37.320447 25403 hybrid_clock.cc:648] HybridClock initialized: now 1790584297320398 us; error 65 us; skew 500 ppm
I20260928 08:31:37.320874 25403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:31:37.323321 25403 webserver.cc:460] Webserver started at http://127.15.134.194:43395/ using document root <none> and password file <none>
I20260928 08:31:37.324076 25403 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:37.324216 25403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:37.324563 25403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:31:37.327651 25403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/data/instance:
uuid: "16efc450c4df41a894e3d44aee5bd8f3"
format_stamp: "Formatted at 2026-09-28 08:31:37 on dist-test-slave-qp2n"
server_key: "3171a5faf0d152b66ccaa5422e2e46fd"
server_key_iv: "1cec76dee8c951dc168ac8dfb9dd5f29"
server_key_version: "encryptionkey@0"
I20260928 08:31:37.328486 25403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance:
uuid: "16efc450c4df41a894e3d44aee5bd8f3"
format_stamp: "Formatted at 2026-09-28 08:31:37 on dist-test-slave-qp2n"
server_key: "3171a5faf0d152b66ccaa5422e2e46fd"
server_key_iv: "1cec76dee8c951dc168ac8dfb9dd5f29"
server_key_version: "encryptionkey@0"
I20260928 08:31:37.334185 25403 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 08:31:37.337936 25417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:37.339370 25403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:31:37.339550 25403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "16efc450c4df41a894e3d44aee5bd8f3"
format_stamp: "Formatted at 2026-09-28 08:31:37 on dist-test-slave-qp2n"
server_key: "3171a5faf0d152b66ccaa5422e2e46fd"
server_key_iv: "1cec76dee8c951dc168ac8dfb9dd5f29"
server_key_version: "encryptionkey@0"
I20260928 08:31:37.339766 25403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:37.354009 25403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:37.354890 25403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:37.355209 25403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:37.356163 25403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:31:37.357789 25403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:31:37.357877 25403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:37.357961 25403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:31:37.358016 25403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:37.389037 25403 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:43527
I20260928 08:31:37.389083 25529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:43527 every 8 connection(s)
I20260928 08:31:37.390372 25403 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:31:37.397717 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 25403
I20260928 08:31:37.397895 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance
I20260928 08:31:37.398355 15899 external_mini_cluster.cc:1428] Setting key 1b5b8fd0dafb789c46e08f6804046cd7
I20260928 08:31:37.401242 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:0
--local_ip_for_outbound_sockets=127.15.134.195
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:42463
--builtin_ntp_servers=127.15.134.212:44761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 08:31:37.403328 25401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:37.393030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:55059 (local address 127.15.134.254:42463)
0928 08:31:37.393208 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:37.393214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:37.394689 (+ 1475us) server_negotiation.cc:407] Connection header received
0928 08:31:37.396410 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:37.396415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:37.396480 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:37.396569 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:37.397710 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:37.398538 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:37.400440 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:37.400905 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:37.401038 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:37.402503 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:37.402509 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:37.402514 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:31:37.402540 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:37.402577 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:31:37.402589 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:37.402683 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:37.402905 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 08:31:37.403005 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:31:37.404381 25532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:37.393600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:42463 (local address 127.15.134.194:55059)
0928 08:31:37.394548 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:37.394590 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:37.396149 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:37.396812 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:37.396820 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:37.396906 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:37.397538 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:37.397554 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:37.398689 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:37.398697 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:37.400285 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:37.400298 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:37.401139 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:37.401147 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:37.401401 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:37.402164 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:31:37.402167 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:37.402212 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:31:37.402302 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:37.402681 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:37.402693 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:37.402703 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 08:31:37.402898 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 08:31:37.403332 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":298,"threads_started":1}
I20260928 08:31:37.405790 25530 heartbeater.cc:347] Connected to a master server at 127.15.134.254:42463
I20260928 08:31:37.406131 25530 heartbeater.cc:464] Registering TS with master...
I20260928 08:31:37.407066 25530 heartbeater.cc:511] Master 127.15.134.254:42463 requested a full tablet report, sending...
I20260928 08:31:37.408692 25213 ts_manager.cc:194] Registered new tserver with Master: 16efc450c4df41a894e3d44aee5bd8f3 (127.15.134.194:43527)
I20260928 08:31:37.409476 25213 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.134.194:55059
W20260928 08:31:37.741858 25533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:37.742161 25533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:37.742323 25533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:37.749945 25533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:37.750063 25533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:37.750234 25533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:31:37.759781 25533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:44761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:42463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:37.761212 25533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:37.763135 25533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:37.777555 25539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:37.777555 25538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:37.777597 25541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:37.778535 25533 server_base.cc:1034] running on GCE node
I20260928 08:31:37.779184 25533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:37.780179 25533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:37.781400 25533 hybrid_clock.cc:648] HybridClock initialized: now 1790584297781338 us; error 68 us; skew 500 ppm
I20260928 08:31:37.781737 25533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:31:37.783950 25533 webserver.cc:460] Webserver started at http://127.15.134.195:39591/ using document root <none> and password file <none>
I20260928 08:31:37.784741 25533 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:37.784873 25533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:37.785239 25533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:31:37.788205 25533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/data/instance:
uuid: "94ad4ee691bd47f99c117512efcbce3b"
format_stamp: "Formatted at 2026-09-28 08:31:37 on dist-test-slave-qp2n"
server_key: "b396978df35d98377bd0d90cd6c0a255"
server_key_iv: "0c49a6851b9bf8be2b54aba20bc11e1c"
server_key_version: "encryptionkey@0"
I20260928 08:31:37.789052 25533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance:
uuid: "94ad4ee691bd47f99c117512efcbce3b"
format_stamp: "Formatted at 2026-09-28 08:31:37 on dist-test-slave-qp2n"
server_key: "b396978df35d98377bd0d90cd6c0a255"
server_key_iv: "0c49a6851b9bf8be2b54aba20bc11e1c"
server_key_version: "encryptionkey@0"
I20260928 08:31:37.795014 25533 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260928 08:31:37.798800 25547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:37.800251 25533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:31:37.800442 25533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "94ad4ee691bd47f99c117512efcbce3b"
format_stamp: "Formatted at 2026-09-28 08:31:37 on dist-test-slave-qp2n"
server_key: "b396978df35d98377bd0d90cd6c0a255"
server_key_iv: "0c49a6851b9bf8be2b54aba20bc11e1c"
server_key_version: "encryptionkey@0"
I20260928 08:31:37.800710 25533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:37.830355 25533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:37.831256 25533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:37.831557 25533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:37.832547 25533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:31:37.834085 25533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:31:37.834208 25533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:37.834340 25533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:31:37.834429 25533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:37.867388 25533 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:39987
I20260928 08:31:37.867470 25659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:39987 every 8 connection(s)
I20260928 08:31:37.868767 25533 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:31:37.877141 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 25533
I20260928 08:31:37.877336 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance
I20260928 08:31:37.877797 15899 external_mini_cluster.cc:1428] Setting key 99bcbda7d977b21d51faf326fcea887f
I20260928 08:31:37.881644 25401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:37.871375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:54717 (local address 127.15.134.254:42463)
0928 08:31:37.871628 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:37.871633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:37.873009 (+ 1376us) server_negotiation.cc:407] Connection header received
0928 08:31:37.874889 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:37.874895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:37.874991 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:37.875104 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:37.876229 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:37.877051 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:37.878891 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:37.879305 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:37.879434 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:37.880821 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:37.880826 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:37.880829 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:31:37.880847 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:37.880880 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:31:37.880892 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:37.880994 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:37.881217 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 08:31:37.881315 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 08:31:37.882684 25662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:37.871903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:42463 (local address 127.15.134.195:54717)
0928 08:31:37.872857 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:37.872900 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:37.874547 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:37.875338 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:37.875347 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:37.875440 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:37.876050 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:37.876076 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:37.877207 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:37.877212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:37.878756 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:37.878767 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:37.879528 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:37.879535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:37.879762 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:37.880457 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:31:37.880460 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:37.880505 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:31:37.880585 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:37.880985 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:37.880996 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:37.881006 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 08:31:37.881211 (+ 205us) client_negotiation.cc:240] Negotiation successful
0928 08:31:37.881610 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":260,"threads_started":1}
I20260928 08:31:37.884084 25660 heartbeater.cc:347] Connected to a master server at 127.15.134.254:42463
I20260928 08:31:37.884403 25660 heartbeater.cc:464] Registering TS with master...
I20260928 08:31:37.885200 25660 heartbeater.cc:511] Master 127.15.134.254:42463 requested a full tablet report, sending...
I20260928 08:31:37.886613 25213 ts_manager.cc:194] Registered new tserver with Master: 94ad4ee691bd47f99c117512efcbce3b (127.15.134.195:39987)
I20260928 08:31:37.887368 25213 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.134.195:54717
I20260928 08:31:37.893055 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 08:31:37.899566 25401 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:37.897811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56992 (local address 127.15.134.254:42463)
0928 08:31:37.898049 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:37.898057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:37.898458 (+ 401us) server_negotiation.cc:407] Connection header received
0928 08:31:37.898672 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:37.898678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:37.898763 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:37.898865 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:37.899289 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56992: BlockingRecv error: recv got EOF from 127.0.0.1:56992 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 08:31:37.900646 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 25272
I20260928 08:31:37.909384 25669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:37.901644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42434 (local address 127.15.134.193:35133)
0928 08:31:37.902359 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:37.902366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:37.902393 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:31:37.902456 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:37.902462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:37.902692 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:37.902820 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:37.903161 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:37.904307 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:37.905735 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:37.906339 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:37.906458 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:37.907765 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:37.907775 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:37.907790 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:31:37.908458 (+ 668us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:37.908511 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:31:37.908537 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:37.908752 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:37.908814 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 08:31:37.908967 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":457,"threads_started":1}
I20260928 08:31:37.910755 25394 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:37.992584 25399 heartbeater.cc:503] Master 127.15.134.254:42463 was elected leader, sending a full tablet report...
I20260928 08:31:37.993928 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 25272
W20260928 08:31:38.013072 25197 connection.cc:535] server connection from 127.15.134.193:35985 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 08:31:38.013373 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 25403
I20260928 08:31:38.021670 25671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:38.014344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43338 (local address 127.15.134.194:43527)
0928 08:31:38.014947 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:38.014956 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:38.014986 (+ 30us) server_negotiation.cc:407] Connection header received
0928 08:31:38.015055 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:38.015062 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:38.015329 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:38.015491 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:38.015886 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:38.017065 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:38.018477 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:38.019032 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:38.019178 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:38.020168 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:38.020179 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:38.020196 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:31:38.020882 (+ 686us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:38.020930 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:31:38.020949 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:38.021077 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:38.021195 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 08:31:38.021322 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":264,"threads_started":1}
I20260928 08:31:38.022914 25525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:38.106143 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 25403
I20260928 08:31:38.123899 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 25533
I20260928 08:31:38.132169 25673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:38.124847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58468 (local address 127.15.134.195:39987)
0928 08:31:38.125484 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:38.125492 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:38.125523 (+ 31us) server_negotiation.cc:407] Connection header received
0928 08:31:38.125584 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:38.125591 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:38.125812 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:38.125941 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:38.126354 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:38.127478 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:38.129055 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:38.129653 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:38.129758 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:38.130818 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:38.130827 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:38.130841 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:31:38.131452 (+ 611us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:38.131501 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:31:38.131520 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:38.131610 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:38.131741 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 08:31:38.131848 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":312,"threads_started":1}
I20260928 08:31:38.133615 25655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:38.215296 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 25533
I20260928 08:31:38.232415 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 25182
I20260928 08:31:38.233835 25243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:38.336232 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 25182
2026-09-28T08:31:38Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2522 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T08:31:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:31:38Z Disabled control of system clock
I20260928 08:31:38.407411 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:40863
--webserver_interface=127.15.134.254
--webserver_port=0
--builtin_ntp_servers=127.15.134.212:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:40863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 08:31:38.743069 25679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:38.743429 25679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:38.743556 25679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:38.750878 25679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:31:38.750993 25679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:38.751030 25679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:38.751080 25679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:31:38.751128 25679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:31:38.760511 25679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:40863
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:40863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:38.761849 25679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:38.763684 25679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:38.775815 25684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:38.775781 25687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:38.775746 25685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:38.776798 25679 server_base.cc:1034] running on GCE node
I20260928 08:31:38.777524 25679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:38.778937 25679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:38.780145 25679 hybrid_clock.cc:648] HybridClock initialized: now 1790584298780108 us; error 31 us; skew 500 ppm
I20260928 08:31:38.780511 25679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:31:38.782783 25679 webserver.cc:460] Webserver started at http://127.15.134.254:34867/ using document root <none> and password file <none>
I20260928 08:31:38.783499 25679 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:38.783608 25679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:38.783946 25679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:31:38.787047 25679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "b89486a734f14807a6a095f0461a80e9"
format_stamp: "Formatted at 2026-09-28 08:31:38 on dist-test-slave-qp2n"
server_key: "490e7fa89ee45697cf5ea830badc0108"
server_key_iv: "ecac77b3b0de8f26e3c4f3dfa14946e6"
server_key_version: "encryptionkey@0"
I20260928 08:31:38.787901 25679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "b89486a734f14807a6a095f0461a80e9"
format_stamp: "Formatted at 2026-09-28 08:31:38 on dist-test-slave-qp2n"
server_key: "490e7fa89ee45697cf5ea830badc0108"
server_key_iv: "ecac77b3b0de8f26e3c4f3dfa14946e6"
server_key_version: "encryptionkey@0"
I20260928 08:31:38.793203 25679 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 08:31:38.796773 25693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:38.798215 25679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:31:38.798399 25679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "b89486a734f14807a6a095f0461a80e9"
format_stamp: "Formatted at 2026-09-28 08:31:38 on dist-test-slave-qp2n"
server_key: "490e7fa89ee45697cf5ea830badc0108"
server_key_iv: "ecac77b3b0de8f26e3c4f3dfa14946e6"
server_key_version: "encryptionkey@0"
I20260928 08:31:38.798624 25679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:38.804711 25679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:38.805534 25679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:38.805862 25679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:38.823418 25679 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:40863
I20260928 08:31:38.823467 25744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:40863 every 8 connection(s)
I20260928 08:31:38.824846 25679 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:31:38.825238 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 25679
I20260928 08:31:38.825450 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:31:38.826273 15899 external_mini_cluster.cc:1428] Setting key 63245582b4ce7cbde574821a90f62b22
I20260928 08:31:38.829205 25745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:38.834736 25748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:38.828331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40904 (local address 127.15.134.254:40863)
0928 08:31:38.829402 (+ 1071us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:38.829412 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:38.829452 (+ 40us) server_negotiation.cc:407] Connection header received
0928 08:31:38.831019 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:38.831048 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:38.831455 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:38.831840 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:38.832995 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:38.833001 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:38.833013 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:31:38.833027 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:38.833122 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:31:38.833142 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:38.833398 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:38.833571 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 08:31:38.833805 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":395,"threads_started":1}
I20260928 08:31:38.840827 25745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b89486a734f14807a6a095f0461a80e9: Bootstrap starting.
I20260928 08:31:38.844035 25745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b89486a734f14807a6a095f0461a80e9: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:38.845134 25745 log.cc:824] T 00000000000000000000000000000000 P b89486a734f14807a6a095f0461a80e9: Log is configured to *not* fsync() on all Append() calls
I20260928 08:31:38.847715 25745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b89486a734f14807a6a095f0461a80e9: No bootstrap required, opened a new log
I20260928 08:31:38.855540 25745 raft_consensus.cc:348] T 00000000000000000000000000000000 P b89486a734f14807a6a095f0461a80e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b89486a734f14807a6a095f0461a80e9" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 40863 } }
I20260928 08:31:38.856043 25745 raft_consensus.cc:374] T 00000000000000000000000000000000 P b89486a734f14807a6a095f0461a80e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:38.856124 25745 raft_consensus.cc:729] T 00000000000000000000000000000000 P b89486a734f14807a6a095f0461a80e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89486a734f14807a6a095f0461a80e9, State: Initialized, Role: FOLLOWER
I20260928 08:31:38.856776 25745 consensus_queue.cc:260] T 00000000000000000000000000000000 P b89486a734f14807a6a095f0461a80e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b89486a734f14807a6a095f0461a80e9" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 40863 } }
I20260928 08:31:38.856993 25745 raft_consensus.cc:388] T 00000000000000000000000000000000 P b89486a734f14807a6a095f0461a80e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:31:38.857093 25745 raft_consensus.cc:482] T 00000000000000000000000000000000 P b89486a734f14807a6a095f0461a80e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:31:38.857228 25745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b89486a734f14807a6a095f0461a80e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:38.859400 25745 raft_consensus.cc:504] T 00000000000000000000000000000000 P b89486a734f14807a6a095f0461a80e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b89486a734f14807a6a095f0461a80e9" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 40863 } }
I20260928 08:31:38.859871 25745 leader_election.cc:302] T 00000000000000000000000000000000 P b89486a734f14807a6a095f0461a80e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b89486a734f14807a6a095f0461a80e9; no voters:
I20260928 08:31:38.860487 25745 leader_election.cc:288] T 00000000000000000000000000000000 P b89486a734f14807a6a095f0461a80e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:31:38.860674 25750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b89486a734f14807a6a095f0461a80e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:31:38.861931 25750 raft_consensus.cc:686] T 00000000000000000000000000000000 P b89486a734f14807a6a095f0461a80e9 [term 1 LEADER]: Becoming Leader. State: Replica: b89486a734f14807a6a095f0461a80e9, State: Running, Role: LEADER
I20260928 08:31:38.862543 25750 consensus_queue.cc:237] T 00000000000000000000000000000000 P b89486a734f14807a6a095f0461a80e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b89486a734f14807a6a095f0461a80e9" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 40863 } }
I20260928 08:31:38.862825 25745 sys_catalog.cc:564] T 00000000000000000000000000000000 P b89486a734f14807a6a095f0461a80e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:31:38.865504 25751 sys_catalog.cc:455] T 00000000000000000000000000000000 P b89486a734f14807a6a095f0461a80e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b89486a734f14807a6a095f0461a80e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b89486a734f14807a6a095f0461a80e9" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 40863 } } }
I20260928 08:31:38.865895 25751 sys_catalog.cc:458] T 00000000000000000000000000000000 P b89486a734f14807a6a095f0461a80e9 [sys.catalog]: This master's current role is: LEADER
I20260928 08:31:38.865819 25752 sys_catalog.cc:455] T 00000000000000000000000000000000 P b89486a734f14807a6a095f0461a80e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b89486a734f14807a6a095f0461a80e9. Latest consensus state: current_term: 1 leader_uuid: "b89486a734f14807a6a095f0461a80e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b89486a734f14807a6a095f0461a80e9" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 40863 } } }
I20260928 08:31:38.866185 25752 sys_catalog.cc:458] T 00000000000000000000000000000000 P b89486a734f14807a6a095f0461a80e9 [sys.catalog]: This master's current role is: LEADER
I20260928 08:31:38.869283 25759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:31:38.873603 25759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:31:38.882288 25759 catalog_manager.cc:1347] Generated new cluster ID: 832442a092b041d8ba28f384876f5a14
I20260928 08:31:38.882418 25759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:31:38.896334 25759 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:31:38.896909 25759 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:31:38.911064 25759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b89486a734f14807a6a095f0461a80e9: Generated new TSK 0
I20260928 08:31:38.912199 25759 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:31:38.919402 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:0
--local_ip_for_outbound_sockets=127.15.134.193
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:40863
--builtin_ntp_servers=127.15.134.212:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 08:31:39.263125 25769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:39.263496 25769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:39.263641 25769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:39.271135 25769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:39.271292 25769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:39.271467 25769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:31:39.281140 25769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:40863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:39.282470 25769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:39.284318 25769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:39.298365 25777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:39.298393 25774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:39.298336 25775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:39.299635 25769 server_base.cc:1034] running on GCE node
I20260928 08:31:39.300348 25769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:39.301606 25769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:39.302830 25769 hybrid_clock.cc:648] HybridClock initialized: now 1790584299302751 us; error 65 us; skew 500 ppm
I20260928 08:31:39.303283 25769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:31:39.305621 25769 webserver.cc:460] Webserver started at http://127.15.134.193:41863/ using document root <none> and password file <none>
I20260928 08:31:39.306392 25769 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:39.306517 25769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:39.306841 25769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:31:39.311439 25769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/data/instance:
uuid: "224fa975b1c9464e9b9a7a311618f23b"
format_stamp: "Formatted at 2026-09-28 08:31:39 on dist-test-slave-qp2n"
server_key: "dd350309aaf070cbd23c9ea73dac1058"
server_key_iv: "a0ffc66892872bdde964316bbfbdad19"
server_key_version: "encryptionkey@0"
I20260928 08:31:39.312263 25769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance:
uuid: "224fa975b1c9464e9b9a7a311618f23b"
format_stamp: "Formatted at 2026-09-28 08:31:39 on dist-test-slave-qp2n"
server_key: "dd350309aaf070cbd23c9ea73dac1058"
server_key_iv: "a0ffc66892872bdde964316bbfbdad19"
server_key_version: "encryptionkey@0"
I20260928 08:31:39.317842 25769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 08:31:39.321491 25783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:39.322876 25769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:31:39.323066 25769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "224fa975b1c9464e9b9a7a311618f23b"
format_stamp: "Formatted at 2026-09-28 08:31:39 on dist-test-slave-qp2n"
server_key: "dd350309aaf070cbd23c9ea73dac1058"
server_key_iv: "a0ffc66892872bdde964316bbfbdad19"
server_key_version: "encryptionkey@0"
I20260928 08:31:39.323287 25769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:39.329056 25769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:39.329846 25769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:39.330112 25769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:39.331028 25769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:31:39.332437 25769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:31:39.332562 25769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:39.332703 25769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:31:39.332808 25769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:39.366210 25769 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:35909
I20260928 08:31:39.366302 25895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:35909 every 8 connection(s)
I20260928 08:31:39.367580 25769 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:31:39.375509 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 25769
I20260928 08:31:39.375774 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance
I20260928 08:31:39.376745 15899 external_mini_cluster.cc:1428] Setting key f71f292380da5ae1f816b48d17863a72
I20260928 08:31:39.377134 25898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:39.370371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:59715 (local address 127.15.134.254:40863)
0928 08:31:39.370858 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:39.370865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:39.372383 (+ 1518us) server_negotiation.cc:407] Connection header received
0928 08:31:39.374303 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:39.374307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:39.374363 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:39.374464 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:39.376286 (+ 1822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:39.376291 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:39.376294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:31:39.376312 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:39.376355 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:31:39.376367 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:39.376482 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:39.376787 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 08:31:39.376871 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":232,"threads_started":1}
I20260928 08:31:39.378252 25899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:39.371076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:40863 (local address 127.15.134.193:59715)
0928 08:31:39.372201 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:39.372249 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:39.374009 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:39.374798 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:39.374808 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:39.374951 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:39.375863 (+ 912us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:31:39.375868 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:39.375926 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:31:39.376060 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:39.376497 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:39.376510 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:39.376521 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 08:31:39.376890 (+ 369us) client_negotiation.cc:240] Negotiation successful
0928 08:31:39.377228 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":408,"threads_started":1}
I20260928 08:31:39.379578 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:0
--local_ip_for_outbound_sockets=127.15.134.194
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:40863
--builtin_ntp_servers=127.15.134.212:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 08:31:39.380064 25896 heartbeater.cc:347] Connected to a master server at 127.15.134.254:40863
I20260928 08:31:39.380396 25896 heartbeater.cc:464] Registering TS with master...
I20260928 08:31:39.381191 25896 heartbeater.cc:511] Master 127.15.134.254:40863 requested a full tablet report, sending...
I20260928 08:31:39.383611 25710 ts_manager.cc:194] Registered new tserver with Master: 224fa975b1c9464e9b9a7a311618f23b (127.15.134.193:35909)
W20260928 08:31:39.735374 25900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:39.735682 25900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:39.735800 25900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:39.743317 25900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:39.743426 25900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:39.743543 25900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:31:39.753129 25900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:40863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:39.754393 25900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:39.756186 25900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:39.769778 25905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:39.769800 25906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:39.769742 25908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:39.770545 25900 server_base.cc:1034] running on GCE node
I20260928 08:31:39.771178 25900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:39.772277 25900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:39.773561 25900 hybrid_clock.cc:648] HybridClock initialized: now 1790584299773504 us; error 50 us; skew 500 ppm
I20260928 08:31:39.773891 25900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:31:39.776093 25900 webserver.cc:460] Webserver started at http://127.15.134.194:41301/ using document root <none> and password file <none>
I20260928 08:31:39.776832 25900 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:39.776922 25900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:39.777261 25900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:31:39.780141 25900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/data/instance:
uuid: "9753e9ff604d4740ace2d32193473bc9"
format_stamp: "Formatted at 2026-09-28 08:31:39 on dist-test-slave-qp2n"
server_key: "0f99eb24a032f6f69355766c05b8a38e"
server_key_iv: "9f36ce510be71c77be18bdd585ef3fac"
server_key_version: "encryptionkey@0"
I20260928 08:31:39.780949 25900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance:
uuid: "9753e9ff604d4740ace2d32193473bc9"
format_stamp: "Formatted at 2026-09-28 08:31:39 on dist-test-slave-qp2n"
server_key: "0f99eb24a032f6f69355766c05b8a38e"
server_key_iv: "9f36ce510be71c77be18bdd585ef3fac"
server_key_version: "encryptionkey@0"
I20260928 08:31:39.786052 25900 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 08:31:39.789968 25914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:39.791410 25900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260928 08:31:39.791611 25900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "9753e9ff604d4740ace2d32193473bc9"
format_stamp: "Formatted at 2026-09-28 08:31:39 on dist-test-slave-qp2n"
server_key: "0f99eb24a032f6f69355766c05b8a38e"
server_key_iv: "9f36ce510be71c77be18bdd585ef3fac"
server_key_version: "encryptionkey@0"
I20260928 08:31:39.791837 25900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:39.797850 25900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:39.798655 25900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:39.798908 25900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:39.799796 25900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:31:39.801242 25900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:31:39.801347 25900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:39.801474 25900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:31:39.801604 25900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:39.834864 25900 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:39111
I20260928 08:31:39.834960 26026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:39111 every 8 connection(s)
I20260928 08:31:39.836275 25900 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:31:39.843297 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 25900
I20260928 08:31:39.843575 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance
I20260928 08:31:39.844017 15899 external_mini_cluster.cc:1428] Setting key 25b3c10e8a18dcdcb97f5c462f9289a4
I20260928 08:31:39.845046 25898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:39.839022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:44735 (local address 127.15.134.254:40863)
0928 08:31:39.839275 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:39.839282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:39.840869 (+ 1587us) server_negotiation.cc:407] Connection header received
0928 08:31:39.842493 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:39.842497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:39.842571 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:39.842666 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:39.844150 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:39.844156 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:39.844161 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:31:39.844181 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:39.844223 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:31:39.844236 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:39.844360 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:39.844657 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 08:31:39.844730 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 08:31:39.845898 26029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:39.839613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:40863 (local address 127.15.134.194:44735)
0928 08:31:39.840703 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:39.840755 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:39.842252 (+ 1497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:39.842894 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:39.842902 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:39.842997 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:39.843791 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:31:39.843794 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:39.843833 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:31:39.843920 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:39.844329 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:39.844340 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:39.844347 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 08:31:39.844665 (+ 318us) client_negotiation.cc:240] Negotiation successful
0928 08:31:39.844925 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":346,"threads_started":1}
I20260928 08:31:39.847216 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:0
--local_ip_for_outbound_sockets=127.15.134.195
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:40863
--builtin_ntp_servers=127.15.134.212:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 08:31:39.847388 26027 heartbeater.cc:347] Connected to a master server at 127.15.134.254:40863
I20260928 08:31:39.847641 26027 heartbeater.cc:464] Registering TS with master...
I20260928 08:31:39.848332 26027 heartbeater.cc:511] Master 127.15.134.254:40863 requested a full tablet report, sending...
I20260928 08:31:39.849934 25710 ts_manager.cc:194] Registered new tserver with Master: 9753e9ff604d4740ace2d32193473bc9 (127.15.134.194:39111)
W20260928 08:31:40.189190 26030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:40.189473 26030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:40.189566 26030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:40.196924 26030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:40.197026 26030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:40.197134 26030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:31:40.206466 26030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:40863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:40.207664 26030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:40.209379 26030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:40.222473 26038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:40.222601 26035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:40.222591 26036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:40.223748 26030 server_base.cc:1034] running on GCE node
I20260928 08:31:40.224380 26030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:40.225427 26030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:40.226673 26030 hybrid_clock.cc:648] HybridClock initialized: now 1790584300226616 us; error 51 us; skew 500 ppm
I20260928 08:31:40.227027 26030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:31:40.229506 26030 webserver.cc:460] Webserver started at http://127.15.134.195:33641/ using document root <none> and password file <none>
I20260928 08:31:40.230302 26030 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:40.230433 26030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:40.230777 26030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:31:40.233920 26030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/data/instance:
uuid: "087ab3e8229e467f81a4eb49326a270b"
format_stamp: "Formatted at 2026-09-28 08:31:40 on dist-test-slave-qp2n"
server_key: "4db8b5fd45d9dc734f09c8342d691cec"
server_key_iv: "f046fc3c151840f50c5c46483a65f8e5"
server_key_version: "encryptionkey@0"
I20260928 08:31:40.234860 26030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance:
uuid: "087ab3e8229e467f81a4eb49326a270b"
format_stamp: "Formatted at 2026-09-28 08:31:40 on dist-test-slave-qp2n"
server_key: "4db8b5fd45d9dc734f09c8342d691cec"
server_key_iv: "f046fc3c151840f50c5c46483a65f8e5"
server_key_version: "encryptionkey@0"
I20260928 08:31:40.240458 26030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 08:31:40.244249 26044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:40.245851 26030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:31:40.246057 26030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "087ab3e8229e467f81a4eb49326a270b"
format_stamp: "Formatted at 2026-09-28 08:31:40 on dist-test-slave-qp2n"
server_key: "4db8b5fd45d9dc734f09c8342d691cec"
server_key_iv: "f046fc3c151840f50c5c46483a65f8e5"
server_key_version: "encryptionkey@0"
I20260928 08:31:40.246306 26030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:40.253237 26030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:40.254133 26030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:40.254427 26030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:40.255386 26030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:31:40.257026 26030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:31:40.257159 26030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:40.257290 26030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:31:40.257377 26030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:40.291651 26030 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:37901
I20260928 08:31:40.291742 26156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:37901 every 8 connection(s)
I20260928 08:31:40.293210 26030 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:31:40.301308 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 26030
I20260928 08:31:40.301522 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance
I20260928 08:31:40.301633 25898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:40.295914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:58511 (local address 127.15.134.254:40863)
0928 08:31:40.296175 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:40.296179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:40.297641 (+ 1462us) server_negotiation.cc:407] Connection header received
0928 08:31:40.299316 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:40.299322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:40.299403 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:40.299528 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:40.300799 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:40.300805 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:40.300810 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:31:40.300829 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:40.300876 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:31:40.300887 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:40.301047 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:40.301250 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 08:31:40.301320 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:31:40.302054 15899 external_mini_cluster.cc:1428] Setting key 67929fd76ff3f6596523e21e074336c6
I20260928 08:31:40.302593 26159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:40.296482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:40863 (local address 127.15.134.195:58511)
0928 08:31:40.297465 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:40.297516 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:40.299054 (+ 1538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:40.299746 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:40.299753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:40.299843 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:40.300470 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:31:40.300473 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:40.300511 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:31:40.300592 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:40.301036 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:40.301047 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:40.301055 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 08:31:40.301333 (+ 278us) client_negotiation.cc:240] Negotiation successful
0928 08:31:40.301585 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":345,"threads_started":1}
I20260928 08:31:40.304069 26157 heartbeater.cc:347] Connected to a master server at 127.15.134.254:40863
I20260928 08:31:40.304355 26157 heartbeater.cc:464] Registering TS with master...
I20260928 08:31:40.305075 26157 heartbeater.cc:511] Master 127.15.134.254:40863 requested a full tablet report, sending...
I20260928 08:31:40.306326 25709 ts_manager.cc:194] Registered new tserver with Master: 087ab3e8229e467f81a4eb49326a270b (127.15.134.195:37901)
I20260928 08:31:40.317368 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 08:31:40.323292 25898 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:31:40.321809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40910 (local address 127.15.134.254:40863)
0928 08:31:40.321970 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:40.321974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:40.322551 (+ 577us) server_negotiation.cc:407] Connection header received
0928 08:31:40.322660 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:40.322664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:40.322717 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:40.322809 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:40.323094 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40910: BlockingRecv error: recv got EOF from 127.0.0.1:40910 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:31:40.323905 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 25769
I20260928 08:31:40.327315 26166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:40.324934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38050 (local address 127.15.134.193:35909)
0928 08:31:40.325391 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:40.325398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:40.325421 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:31:40.325474 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:40.325480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:40.325677 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:40.325799 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:40.326688 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:40.326696 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:40.326707 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:31:40.326720 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:40.326764 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:31:40.326778 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:40.326916 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:40.326977 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 08:31:40.327060 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":243,"threads_started":1}
I20260928 08:31:40.328562 25891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:40.408890 25896 heartbeater.cc:503] Master 127.15.134.254:40863 was elected leader, sending a full tablet report...
I20260928 08:31:40.410589 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 25769
W20260928 08:31:40.431699 25697 connection.cc:535] server connection from 127.15.134.193:59715 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 08:31:40.432065 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 25900
I20260928 08:31:40.435763 26168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:40.433047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43030 (local address 127.15.134.194:39111)
0928 08:31:40.433669 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:40.433676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:40.433705 (+ 29us) server_negotiation.cc:407] Connection header received
0928 08:31:40.433767 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:40.433773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:40.433998 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:40.434135 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:40.435116 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:40.435122 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:40.435135 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:31:40.435148 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:40.435217 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:31:40.435232 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:40.435340 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:40.435437 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 08:31:40.435520 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":276,"threads_started":1}
I20260928 08:31:40.437069 26022 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:40.518162 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 25900
I20260928 08:31:40.534876 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 26030
I20260928 08:31:40.538938 26170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:40.535895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48784 (local address 127.15.134.195:37901)
0928 08:31:40.536592 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:40.536599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:40.536678 (+ 79us) server_negotiation.cc:407] Connection header received
0928 08:31:40.536755 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:40.536762 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:40.536992 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:40.537123 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:40.538104 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:40.538110 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:40.538122 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:31:40.538134 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:40.538176 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:31:40.538192 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:40.538324 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:40.538584 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 08:31:40.538676 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":363,"threads_started":1}
I20260928 08:31:40.540211 26152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:40.623255 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 26030
I20260928 08:31:40.640558 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 25679
I20260928 08:31:40.641908 25740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:40.747001 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 25679
2026-09-28T08:31:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2414 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:31:40 dist-test-slave-qp2n krb5kdc[26175](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:31:40 dist-test-slave-qp2n krb5kdc[26175](info): set up 2 sockets
Sep 28 08:31:40 dist-test-slave-qp2n krb5kdc[26175](info): commencing operation
krb5kdc: starting...
W20260928 08:31:41.863189 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:31:42 dist-test-slave-qp2n krb5kdc[26175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:31:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:31:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:31:42.104218 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:40203
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:35633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:40203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:31:42.443776 26188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:42.444067 26188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:42.444155 26188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:42.451270 26188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:31:42.451367 26188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:42.451411 26188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:42.451524 26188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:31:42.451571 26188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:31:42.461154 26188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:35633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:40203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:40203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:42.462522 26188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:42.463780 26188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:42.475843 26194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:42.476033 26193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:42.475798 26196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:42.476923 26188 server_base.cc:1034] running on GCE node
I20260928 08:31:42.477789 26188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:42.479346 26188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:42.480928 26188 hybrid_clock.cc:648] HybridClock initialized: now 1790584302480598 us; error 42 us; skew 500 ppm
I20260928 08:31:42.481336 26188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:31:42 dist-test-slave-qp2n krb5kdc[26175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584302, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:31:42.489817 26188 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:31:42.491552 26188 webserver.cc:460] Webserver started at http://127.15.134.254:46671/ using document root <none> and password file <none>
I20260928 08:31:42.492316 26188 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:42.492471 26188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:42.492849 26188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:31:42.496074 26188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "33e693c83f814687928f0939290596e4"
format_stamp: "Formatted at 2026-09-28 08:31:42 on dist-test-slave-qp2n"
server_key: "0689440ce9b65a49dbb285d1c5a373aa"
server_key_iv: "976e0c8f56400d1e8dbb777c1e955fc8"
server_key_version: "encryptionkey@0"
I20260928 08:31:42.497009 26188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "33e693c83f814687928f0939290596e4"
format_stamp: "Formatted at 2026-09-28 08:31:42 on dist-test-slave-qp2n"
server_key: "0689440ce9b65a49dbb285d1c5a373aa"
server_key_iv: "976e0c8f56400d1e8dbb777c1e955fc8"
server_key_version: "encryptionkey@0"
I20260928 08:31:42.502492 26188 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:31:42.506338 26203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:42.508082 26188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:31:42.508283 26188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "33e693c83f814687928f0939290596e4"
format_stamp: "Formatted at 2026-09-28 08:31:42 on dist-test-slave-qp2n"
server_key: "0689440ce9b65a49dbb285d1c5a373aa"
server_key_iv: "976e0c8f56400d1e8dbb777c1e955fc8"
server_key_version: "encryptionkey@0"
I20260928 08:31:42.508533 26188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:42.535029 26188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:42.539448 26188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:42.539798 26188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:42.559975 26188 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:40203
I20260928 08:31:42.559999 26254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:40203 every 8 connection(s)
I20260928 08:31:42.562466 26188 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:31:42.568549 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 26188
I20260928 08:31:42.568873 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:31:42.568985 26255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:42.569849 15899 external_mini_cluster.cc:1428] Setting key 2ca36e26c39c7063f198affbef895980
I20260928 08:31:42.589797 26255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33e693c83f814687928f0939290596e4: Bootstrap starting.
Sep 28 08:31:42 dist-test-slave-qp2n krb5kdc[26175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:31:42.595005 26255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33e693c83f814687928f0939290596e4: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:42.596969 26255 log.cc:824] T 00000000000000000000000000000000 P 33e693c83f814687928f0939290596e4: Log is configured to *not* fsync() on all Append() calls
I20260928 08:31:42.601434 26255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33e693c83f814687928f0939290596e4: No bootstrap required, opened a new log
I20260928 08:31:42.602502 26258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:42.574304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33092 (local address 127.15.134.254:40203)
0928 08:31:42.575409 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:42.575433 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:42.575495 (+ 62us) server_negotiation.cc:407] Connection header received
0928 08:31:42.578420 (+ 2925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:42.578455 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:42.579127 (+ 672us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:42.579689 (+ 562us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:42.581754 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:42.583794 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:42.585456 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:42.586331 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:42.586583 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:42.592519 (+ 5936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:42.592532 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:42.592554 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:42.592664 (+ 110us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:42.597226 (+ 4562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:42.598096 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:42.598106 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:42.598115 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:42.598399 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:42.598794 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:42.598799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:42.598802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:42.599692 (+ 890us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:42.600092 (+ 400us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:42.600860 (+ 768us) server_negotiation.cc:299] Negotiation successful
0928 08:31:42.601442 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":808,"thread_start_us":547,"threads_started":1}
I20260928 08:31:42.616811 26255 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33e693c83f814687928f0939290596e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e693c83f814687928f0939290596e4" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 40203 } }
I20260928 08:31:42.617486 26255 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33e693c83f814687928f0939290596e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:42.617617 26255 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33e693c83f814687928f0939290596e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33e693c83f814687928f0939290596e4, State: Initialized, Role: FOLLOWER
I20260928 08:31:42.618425 26255 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33e693c83f814687928f0939290596e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e693c83f814687928f0939290596e4" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 40203 } }
I20260928 08:31:42.618795 26255 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33e693c83f814687928f0939290596e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:31:42.618969 26255 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33e693c83f814687928f0939290596e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:31:42.619170 26255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33e693c83f814687928f0939290596e4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:42.622195 26255 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33e693c83f814687928f0939290596e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e693c83f814687928f0939290596e4" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 40203 } }
I20260928 08:31:42.622697 26255 leader_election.cc:302] T 00000000000000000000000000000000 P 33e693c83f814687928f0939290596e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33e693c83f814687928f0939290596e4; no voters:
I20260928 08:31:42.623270 26255 leader_election.cc:288] T 00000000000000000000000000000000 P 33e693c83f814687928f0939290596e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:31:42.623435 26260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33e693c83f814687928f0939290596e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:31:42.624151 26260 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33e693c83f814687928f0939290596e4 [term 1 LEADER]: Becoming Leader. State: Replica: 33e693c83f814687928f0939290596e4, State: Running, Role: LEADER
I20260928 08:31:42.624795 26260 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33e693c83f814687928f0939290596e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e693c83f814687928f0939290596e4" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 40203 } }
I20260928 08:31:42.625641 26255 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33e693c83f814687928f0939290596e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:31:42.627717 26261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33e693c83f814687928f0939290596e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33e693c83f814687928f0939290596e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e693c83f814687928f0939290596e4" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 40203 } } }
I20260928 08:31:42.628225 26261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33e693c83f814687928f0939290596e4 [sys.catalog]: This master's current role is: LEADER
I20260928 08:31:42.629546 26267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:31:42.629546 26262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33e693c83f814687928f0939290596e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33e693c83f814687928f0939290596e4. Latest consensus state: current_term: 1 leader_uuid: "33e693c83f814687928f0939290596e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33e693c83f814687928f0939290596e4" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 40203 } } }
I20260928 08:31:42.630110 26262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33e693c83f814687928f0939290596e4 [sys.catalog]: This master's current role is: LEADER
I20260928 08:31:42.635377 26267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:31:42.644816 26267 catalog_manager.cc:1347] Generated new cluster ID: 6d09e0f18fa34a2997225193eb21e5f0
I20260928 08:31:42.644955 26267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:31:42.659170 26267 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:31:42.660548 26267 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:31:42.667228 26267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33e693c83f814687928f0939290596e4: Generated new TSK 0
I20260928 08:31:42.668149 26267 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.134.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
I20260928 08:31:42.725257 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:0
--local_ip_for_outbound_sockets=127.15.134.193
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:40203
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:35633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:31:43.062865 26281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:43.063179 26281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:43.063273 26281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:43.070526 26281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:43.070685 26281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:43.070865 26281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:31:43.080724 26281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:35633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:40203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:43.081997 26281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:43.083338 26281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:43.096913 26286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:43.096946 26287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:43.096844 26289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:43.097911 26281 server_base.cc:1034] running on GCE node
I20260928 08:31:43.098580 26281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:43.099606 26281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:43.100812 26281 hybrid_clock.cc:648] HybridClock initialized: now 1790584303100769 us; error 109 us; skew 500 ppm
I20260928 08:31:43.101140 26281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:31:43 dist-test-slave-qp2n krb5kdc[26175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584303, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:31:43.104358 26281 init.cc:377] Logged in from keytab as kudu/127.15.134.193@KRBTEST.COM (short username kudu)
I20260928 08:31:43.105917 26281 webserver.cc:460] Webserver started at http://127.15.134.193:45875/ using document root <none> and password file <none>
I20260928 08:31:43.106624 26281 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:43.106773 26281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:43.107090 26281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:31:43.110168 26281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/data/instance:
uuid: "6092e2b6d0d64fa98e2f638fcc63b447"
format_stamp: "Formatted at 2026-09-28 08:31:43 on dist-test-slave-qp2n"
server_key: "7481d4ce2be33fe9f23d65e5f984cffb"
server_key_iv: "e0842f2750d413017b0cf77c96abb148"
server_key_version: "encryptionkey@0"
I20260928 08:31:43.111019 26281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance:
uuid: "6092e2b6d0d64fa98e2f638fcc63b447"
format_stamp: "Formatted at 2026-09-28 08:31:43 on dist-test-slave-qp2n"
server_key: "7481d4ce2be33fe9f23d65e5f984cffb"
server_key_iv: "e0842f2750d413017b0cf77c96abb148"
server_key_version: "encryptionkey@0"
I20260928 08:31:43.116590 26281 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 08:31:43.120157 26296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:43.121572 26281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 08:31:43.121753 26281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "6092e2b6d0d64fa98e2f638fcc63b447"
format_stamp: "Formatted at 2026-09-28 08:31:43 on dist-test-slave-qp2n"
server_key: "7481d4ce2be33fe9f23d65e5f984cffb"
server_key_iv: "e0842f2750d413017b0cf77c96abb148"
server_key_version: "encryptionkey@0"
I20260928 08:31:43.121955 26281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:43.134999 26281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:43.139212 26281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:43.139521 26281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:43.140478 26281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:31:43.141969 26281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:31:43.142071 26281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:43.142215 26281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:31:43.142295 26281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:43.174342 26281 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:41421
I20260928 08:31:43.174386 26408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:41421 every 8 connection(s)
I20260928 08:31:43.175638 26281 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:31:43.179505 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 26281
I20260928 08:31:43.179744 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance
I20260928 08:31:43.180447 15899 external_mini_cluster.cc:1428] Setting key 5eabfee401c915c3d8174fcfd3aee5d1
Sep 28 08:31:43 dist-test-slave-qp2n krb5kdc[26175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584303, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:31:43.197809 26411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:43.178352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:35413 (local address 127.15.134.254:40203)
0928 08:31:43.178802 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:43.178806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:43.179879 (+ 1073us) server_negotiation.cc:407] Connection header received
0928 08:31:43.181990 (+ 2111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:43.181994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:43.182057 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:43.182151 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:43.183780 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:43.184472 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:43.186120 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:43.186514 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:43.186651 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:43.190288 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:43.190298 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:43.190306 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:43.190352 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:43.193909 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:43.194881 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:43.194889 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:43.194895 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:43.195056 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:43.195583 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:43.195591 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:43.195597 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:43.195980 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:43.196147 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:43.197064 (+ 917us) server_negotiation.cc:299] Negotiation successful
0928 08:31:43.197329 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":247,"threads_started":1}
I20260928 08:31:43.198578 26412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:43.178812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:40203 (local address 127.15.134.193:35413)
0928 08:31:43.179649 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:43.179719 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:43.181701 (+ 1982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:43.182456 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:43.182486 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:43.183030 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:43.183598 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:43.183615 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:43.184669 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:43.184673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:43.185947 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:43.185960 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:43.186743 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:43.186750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:43.186988 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:43.187676 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:43.187699 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:43.190058 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:43.194110 (+ 4052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:43.194118 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:43.194133 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:43.194726 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:43.195171 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:43.195174 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:43.195177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:43.195450 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:43.196125 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:43.196132 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:43.196560 (+ 428us) client_negotiation.cc:769] Sending connection context
0928 08:31:43.196900 (+ 340us) client_negotiation.cc:240] Negotiation successful
0928 08:31:43.197337 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":329,"threads_started":1}
I20260928 08:31:43.200341 26409 heartbeater.cc:347] Connected to a master server at 127.15.134.254:40203
I20260928 08:31:43.200753 26409 heartbeater.cc:464] Registering TS with master...
I20260928 08:31:43.201545 26409 heartbeater.cc:511] Master 127.15.134.254:40203 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.134.194@KRBTEST.COM; defaulting to no policy
I20260928 08:31:43.203838 26220 ts_manager.cc:194] Registered new tserver with Master: 6092e2b6d0d64fa98e2f638fcc63b447 (127.15.134.193:41421)
I20260928 08:31:43.205570 26220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.193@KRBTEST.COM'} at 127.15.134.193:35413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
I20260928 08:31:43.232409 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:0
--local_ip_for_outbound_sockets=127.15.134.194
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:40203
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:35633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:31:43.563134 26415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:43.563524 26415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:43.563683 26415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:43.571085 26415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:43.571245 26415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:43.571425 26415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:31:43.581305 26415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:35633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:40203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:43.582623 26415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:43.584012 26415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:43.597476 26421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:43.597455 26420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:43.597787 26423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:43.598326 26415 server_base.cc:1034] running on GCE node
I20260928 08:31:43.599131 26415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:43.600142 26415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:43.601461 26415 hybrid_clock.cc:648] HybridClock initialized: now 1790584303601418 us; error 49 us; skew 500 ppm
I20260928 08:31:43.601811 26415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:31:43 dist-test-slave-qp2n krb5kdc[26175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584303, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:31:43.605199 26415 init.cc:377] Logged in from keytab as kudu/127.15.134.194@KRBTEST.COM (short username kudu)
I20260928 08:31:43.606920 26415 webserver.cc:460] Webserver started at http://127.15.134.194:36737/ using document root <none> and password file <none>
I20260928 08:31:43.607671 26415 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:43.607831 26415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:43.608219 26415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:31:43.611423 26415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/data/instance:
uuid: "14d275ba698146768378aa019bbe3930"
format_stamp: "Formatted at 2026-09-28 08:31:43 on dist-test-slave-qp2n"
server_key: "99fa724870dac249d27bcb2a73dfea49"
server_key_iv: "f4e75f51aa1b41dd2bd9ef0ff1e4ddfb"
server_key_version: "encryptionkey@0"
I20260928 08:31:43.612286 26415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance:
uuid: "14d275ba698146768378aa019bbe3930"
format_stamp: "Formatted at 2026-09-28 08:31:43 on dist-test-slave-qp2n"
server_key: "99fa724870dac249d27bcb2a73dfea49"
server_key_iv: "f4e75f51aa1b41dd2bd9ef0ff1e4ddfb"
server_key_version: "encryptionkey@0"
I20260928 08:31:43.618487 26415 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260928 08:31:43.622284 26430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:43.623816 26415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:31:43.623996 26415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "14d275ba698146768378aa019bbe3930"
format_stamp: "Formatted at 2026-09-28 08:31:43 on dist-test-slave-qp2n"
server_key: "99fa724870dac249d27bcb2a73dfea49"
server_key_iv: "f4e75f51aa1b41dd2bd9ef0ff1e4ddfb"
server_key_version: "encryptionkey@0"
I20260928 08:31:43.624207 26415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:43.642109 26415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:43.646744 26415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:43.647075 26415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:43.648126 26415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:31:43.649629 26415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:31:43.649740 26415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:43.649899 26415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:31:43.649982 26415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:43.687075 26415 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:41625
I20260928 08:31:43.687114 26542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:41625 every 8 connection(s)
I20260928 08:31:43.688377 26415 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:31:43.696005 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 26415
I20260928 08:31:43.696349 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance
I20260928 08:31:43.696940 15899 external_mini_cluster.cc:1428] Setting key b3d058625af0e863f851e10059f5c063
Sep 28 08:31:43 dist-test-slave-qp2n krb5kdc[26175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584303, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:31:43.711124 26411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:43.691051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:38111 (local address 127.15.134.254:40203)
0928 08:31:43.691259 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:43.691265 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:43.692624 (+ 1359us) server_negotiation.cc:407] Connection header received
0928 08:31:43.694461 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:43.694466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:43.694539 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:43.694635 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:43.696296 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:43.697323 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:43.699428 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:43.700197 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:43.700342 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:43.704302 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:43.704312 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:43.704319 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:43.704367 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:43.707512 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:43.708532 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:43.708535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:43.708537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:43.708729 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:43.709268 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:43.709271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:43.709273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:43.709541 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:43.709651 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:43.710654 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0928 08:31:43.710842 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:31:43.712991 26545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:43.691661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:40203 (local address 127.15.134.194:38111)
0928 08:31:43.692393 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:43.692439 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:43.694167 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:43.694881 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:43.694890 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:43.695427 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:43.696062 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:43.696081 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:43.697585 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:43.697593 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:43.699253 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:43.699274 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:43.700444 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:43.700452 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:43.700716 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:43.701453 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:43.701477 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:43.704061 (+ 2584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:43.707664 (+ 3603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:43.707677 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:43.707698 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:43.708388 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:43.708852 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:43.708860 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:43.708867 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:43.709142 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:43.709702 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:43.709713 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:43.710222 (+ 509us) client_negotiation.cc:769] Sending connection context
0928 08:31:43.710569 (+ 347us) client_negotiation.cc:240] Negotiation successful
0928 08:31:43.711304 (+ 735us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":288,"threads_started":1}
I20260928 08:31:43.714890 26543 heartbeater.cc:347] Connected to a master server at 127.15.134.254:40203
I20260928 08:31:43.715260 26543 heartbeater.cc:464] Registering TS with master...
I20260928 08:31:43.716231 26543 heartbeater.cc:511] Master 127.15.134.254:40203 requested a full tablet report, sending...
I20260928 08:31:43.717989 26220 ts_manager.cc:194] Registered new tserver with Master: 14d275ba698146768378aa019bbe3930 (127.15.134.194:41625)
I20260928 08:31:43.718811 26220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.194@KRBTEST.COM'} at 127.15.134.194:38111
WARNING: no policy specified for kudu/127.15.134.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
I20260928 08:31:43.757483 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:0
--local_ip_for_outbound_sockets=127.15.134.195
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:40203
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:35633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:31:44.093117 26548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:31:44.093394 26548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:31:44.093547 26548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:31:44.101071 26548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:31:44.101179 26548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:31:44.101372 26548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:31:44.111370 26548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:35633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:40203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:31:44.112740 26548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:31:44.114109 26548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:31:44.127691 26553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:44.127823 26554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:31:44.128137 26556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:31:44.129518 26548 server_base.cc:1034] running on GCE node
I20260928 08:31:44.130169 26548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:31:44.131090 26548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:31:44.132304 26548 hybrid_clock.cc:648] HybridClock initialized: now 1790584304132259 us; error 34 us; skew 500 ppm
I20260928 08:31:44.132779 26548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:31:44 dist-test-slave-qp2n krb5kdc[26175](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584304, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:31:44.136251 26548 init.cc:377] Logged in from keytab as kudu/127.15.134.195@KRBTEST.COM (short username kudu)
I20260928 08:31:44.138160 26548 webserver.cc:460] Webserver started at http://127.15.134.195:35193/ using document root <none> and password file <none>
I20260928 08:31:44.138904 26548 fs_manager.cc:359] Metadata directory not provided
I20260928 08:31:44.139051 26548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:31:44.139354 26548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:31:44.142656 26548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/data/instance:
uuid: "0c7501ed02814400b9108398fe6aee29"
format_stamp: "Formatted at 2026-09-28 08:31:44 on dist-test-slave-qp2n"
server_key: "9940274bd2984444c678c43c65cbd6ea"
server_key_iv: "3f8a96a2fdad286f2fe60b4513bc3ed5"
server_key_version: "encryptionkey@0"
I20260928 08:31:44.143545 26548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance:
uuid: "0c7501ed02814400b9108398fe6aee29"
format_stamp: "Formatted at 2026-09-28 08:31:44 on dist-test-slave-qp2n"
server_key: "9940274bd2984444c678c43c65cbd6ea"
server_key_iv: "3f8a96a2fdad286f2fe60b4513bc3ed5"
server_key_version: "encryptionkey@0"
I20260928 08:31:44.149088 26548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260928 08:31:44.152588 26563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:44.154032 26548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:31:44.154237 26548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "0c7501ed02814400b9108398fe6aee29"
format_stamp: "Formatted at 2026-09-28 08:31:44 on dist-test-slave-qp2n"
server_key: "9940274bd2984444c678c43c65cbd6ea"
server_key_iv: "3f8a96a2fdad286f2fe60b4513bc3ed5"
server_key_version: "encryptionkey@0"
I20260928 08:31:44.154536 26548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:31:44.166916 26548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:31:44.171178 26548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:31:44.171487 26548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:31:44.172502 26548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:31:44.174190 26548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:31:44.174331 26548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 08:31:44.174510 26548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:31:44.174616 26548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:31:44.208670 26409 heartbeater.cc:503] Master 127.15.134.254:40203 was elected leader, sending a full tablet report...
I20260928 08:31:44.210238 26548 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:43791
I20260928 08:31:44.210240 26675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:43791 every 8 connection(s)
I20260928 08:31:44.211680 26548 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:31:44.221765 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 26548
I20260928 08:31:44.221987 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance
I20260928 08:31:44.222491 15899 external_mini_cluster.cc:1428] Setting key b36a0d61f8b26e6eec52ee164fe1fcc0
Sep 28 08:31:44 dist-test-slave-qp2n krb5kdc[26175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584304, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:31:44.232580 26678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:44.214287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:57645 (local address 127.15.134.254:40203)
0928 08:31:44.214747 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:44.214751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:44.215722 (+ 971us) server_negotiation.cc:407] Connection header received
0928 08:31:44.217608 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:44.217613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:44.217701 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:44.217817 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:44.219289 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.220068 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.221710 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.222210 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.222338 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:44.226430 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:44.226434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:44.226438 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:44.226511 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:44.229586 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:44.230312 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:44.230317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:44.230319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:44.230505 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:44.230949 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:44.230955 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:44.230958 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:44.231261 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:44.231385 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:44.232064 (+ 679us) server_negotiation.cc:299] Negotiation successful
0928 08:31:44.232241 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":272,"threads_started":1}
I20260928 08:31:44.233597 26679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:44.214777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:40203 (local address 127.15.134.195:57645)
0928 08:31:44.215560 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:44.215598 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:44.217343 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:44.218051 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:44.218059 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:44.218569 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:31:44.219125 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:44.219143 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.220226 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.220232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:44.221556 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:44.221572 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.222455 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.222463 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:44.222735 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:44.223486 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:31:44.223525 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:31:44.226243 (+ 2718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:31:44.229709 (+ 3466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:44.229716 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:44.229731 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:44.230176 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:44.230612 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:31:44.230615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:31:44.230617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:31:44.230796 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:31:44.231375 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:31:44.231381 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:31:44.231709 (+ 328us) client_negotiation.cc:769] Sending connection context
0928 08:31:44.231945 (+ 236us) client_negotiation.cc:240] Negotiation successful
0928 08:31:44.232345 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":310,"threads_started":1}
I20260928 08:31:44.235117 26676 heartbeater.cc:347] Connected to a master server at 127.15.134.254:40203
I20260928 08:31:44.235426 26676 heartbeater.cc:464] Registering TS with master...
I20260928 08:31:44.236286 26676 heartbeater.cc:511] Master 127.15.134.254:40203 requested a full tablet report, sending...
I20260928 08:31:44.238088 26220 ts_manager.cc:194] Registered new tserver with Master: 0c7501ed02814400b9108398fe6aee29 (127.15.134.195:43791)
I20260928 08:31:44.238963 26220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.195@KRBTEST.COM'} at 127.15.134.195:57645
I20260928 08:31:44.256451 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:31:44.279381 26678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:44.262897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33108 (local address 127.15.134.254:40203)
0928 08:31:44.263141 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:44.263151 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:44.263593 (+ 442us) server_negotiation.cc:407] Connection header received
0928 08:31:44.263758 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:44.263766 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:44.263873 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:44.264002 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:44.265634 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.267009 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.268441 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.269295 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.269468 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:44.271878 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:44.271887 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:44.271895 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:44.271946 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:44.276069 (+ 4123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:44.276896 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:44.276905 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:44.276911 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:44.277091 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:44.277468 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:44.277475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:44.277480 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:44.277999 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:44.278303 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:44.278635 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 08:31:44.278928 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 08:31:44.284504 26220 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:31:44.298599 26220 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:31:44.342728 26691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:44.312057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:41625 (local address 127.0.0.1:55908)
0928 08:31:44.326873 (+ 14816us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:44.326919 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:44.327121 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:44.330374 (+ 3253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:44.330384 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:44.330422 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:44.330889 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:44.330905 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.333220 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.333228 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:44.337393 (+ 4165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:44.337415 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.341391 (+ 3976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.341400 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:44.341683 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:44.341759 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 08:31:44.341922 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 08:31:44.342192 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14505,"thread_start_us":20698,"threads_started":1}
I20260928 08:31:44.343971 26689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:44.312329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:41421 (local address 127.0.0.1:40978)
0928 08:31:44.333040 (+ 20711us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:44.333062 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:44.333180 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:44.334384 (+ 1204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:44.334392 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:44.334427 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:44.334930 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:44.334944 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.337796 (+ 2852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.337805 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:44.340411 (+ 2606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:44.340431 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.342854 (+ 2423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.342863 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:44.343126 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:44.343180 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 08:31:44.343254 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 08:31:44.343434 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20563,"thread_start_us":14054,"threads_started":1}
I20260928 08:31:44.344550 26690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:44.312300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55908 (local address 127.15.134.194:41625)
0928 08:31:44.327756 (+ 15456us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:44.327768 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:44.327802 (+ 34us) server_negotiation.cc:407] Connection header received
0928 08:31:44.327907 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:44.327918 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:44.328474 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:44.328772 (+ 298us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:44.331105 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.333027 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.337646 (+ 4619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.339712 (+ 2066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.339904 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:44.343203 (+ 3299us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:44.343301 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 08:31:44.343531 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15255,"thread_start_us":15832,"threads_started":1}
I20260928 08:31:44.345933 26692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:44.313122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40978 (local address 127.15.134.193:41421)
0928 08:31:44.332341 (+ 19219us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:44.332353 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:44.333337 (+ 984us) server_negotiation.cc:407] Connection header received
0928 08:31:44.333451 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:44.333459 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:44.333827 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:44.334064 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:44.335216 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.337551 (+ 2335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.340680 (+ 3129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.342588 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.344962 (+ 2374us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:44.345184 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:44.345253 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 08:31:44.345466 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18986,"thread_start_us":19231,"threads_started":1}
I20260928 08:31:44.349622 26478 tablet_service.cc:1467] Processing CreateTablet for tablet 4f72955259d44739839853ae5b35b8df (DEFAULT_TABLE table=test-table [id=e256afa161eb47bcbc69e18d1336ed6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:31:44.351231 26478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f72955259d44739839853ae5b35b8df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:44.351296 26344 tablet_service.cc:1467] Processing CreateTablet for tablet 4f72955259d44739839853ae5b35b8df (DEFAULT_TABLE table=test-table [id=e256afa161eb47bcbc69e18d1336ed6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:31:44.352860 26344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f72955259d44739839853ae5b35b8df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:44.358633 26691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:44.339441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:43791 (local address 127.0.0.1:39144)
0928 08:31:44.344115 (+ 4674us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:44.344139 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:44.344252 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:44.348653 (+ 4401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:44.348661 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:44.348695 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:44.348912 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:44.348926 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.351338 (+ 2412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.351347 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:44.353721 (+ 2374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:44.353742 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.356110 (+ 2368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.356119 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:44.356368 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:44.356424 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 08:31:44.356514 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 08:31:44.358221 (+ 1707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4545,"thread_start_us":5627,"threads_started":1}
I20260928 08:31:44.360069 26693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:44.333489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39144 (local address 127.15.134.195:43791)
0928 08:31:44.346756 (+ 13267us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:44.346764 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:44.346799 (+ 35us) server_negotiation.cc:407] Connection header received
0928 08:31:44.346919 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:44.346927 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:44.347190 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:44.347394 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:44.349143 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.351148 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.353996 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.355871 (+ 1875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.359063 (+ 3192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:44.359328 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:44.359406 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 08:31:44.359628 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13048,"thread_start_us":23860,"threads_started":1}
I20260928 08:31:44.364382 26611 tablet_service.cc:1467] Processing CreateTablet for tablet 4f72955259d44739839853ae5b35b8df (DEFAULT_TABLE table=test-table [id=e256afa161eb47bcbc69e18d1336ed6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:31:44.366164 26611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f72955259d44739839853ae5b35b8df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:31:44.374100 26695 tablet_bootstrap.cc:492] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447: Bootstrap starting.
I20260928 08:31:44.374100 26696 tablet_bootstrap.cc:492] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930: Bootstrap starting.
I20260928 08:31:44.378036 26695 tablet_bootstrap.cc:654] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:44.378027 26696 tablet_bootstrap.cc:654] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:44.379451 26696 log.cc:824] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930: Log is configured to *not* fsync() on all Append() calls
I20260928 08:31:44.379453 26695 log.cc:824] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447: Log is configured to *not* fsync() on all Append() calls
I20260928 08:31:44.384806 26696 tablet_bootstrap.cc:492] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930: No bootstrap required, opened a new log
I20260928 08:31:44.384806 26695 tablet_bootstrap.cc:492] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447: No bootstrap required, opened a new log
I20260928 08:31:44.385118 26695 ts_tablet_manager.cc:1397] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260928 08:31:44.385118 26696 ts_tablet_manager.cc:1397] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260928 08:31:44.396591 26696 raft_consensus.cc:348] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d275ba698146768378aa019bbe3930" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 41625 } } peers { permanent_uuid: "6092e2b6d0d64fa98e2f638fcc63b447" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 41421 } } peers { permanent_uuid: "0c7501ed02814400b9108398fe6aee29" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 43791 } }
I20260928 08:31:44.396596 26695 raft_consensus.cc:348] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d275ba698146768378aa019bbe3930" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 41625 } } peers { permanent_uuid: "6092e2b6d0d64fa98e2f638fcc63b447" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 41421 } } peers { permanent_uuid: "0c7501ed02814400b9108398fe6aee29" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 43791 } }
I20260928 08:31:44.397325 26696 raft_consensus.cc:374] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:44.397334 26695 raft_consensus.cc:374] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:44.397454 26696 raft_consensus.cc:729] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d275ba698146768378aa019bbe3930, State: Initialized, Role: FOLLOWER
I20260928 08:31:44.397560 26695 raft_consensus.cc:729] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6092e2b6d0d64fa98e2f638fcc63b447, State: Initialized, Role: FOLLOWER
I20260928 08:31:44.398317 26696 consensus_queue.cc:260] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d275ba698146768378aa019bbe3930" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 41625 } } peers { permanent_uuid: "6092e2b6d0d64fa98e2f638fcc63b447" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 41421 } } peers { permanent_uuid: "0c7501ed02814400b9108398fe6aee29" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 43791 } }
I20260928 08:31:44.398325 26695 consensus_queue.cc:260] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d275ba698146768378aa019bbe3930" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 41625 } } peers { permanent_uuid: "6092e2b6d0d64fa98e2f638fcc63b447" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 41421 } } peers { permanent_uuid: "0c7501ed02814400b9108398fe6aee29" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 43791 } }
I20260928 08:31:44.401463 26699 tablet_bootstrap.cc:492] T 4f72955259d44739839853ae5b35b8df P 0c7501ed02814400b9108398fe6aee29: Bootstrap starting.
I20260928 08:31:44.406177 26699 tablet_bootstrap.cc:654] T 4f72955259d44739839853ae5b35b8df P 0c7501ed02814400b9108398fe6aee29: Neither blocks nor log segments found. Creating new log.
I20260928 08:31:44.407069 26696 ts_tablet_manager.cc:1428] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930: Time spent starting tablet: real 0.022s user 0.008s sys 0.013s
I20260928 08:31:44.407069 26695 ts_tablet_manager.cc:1428] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447: Time spent starting tablet: real 0.022s user 0.016s sys 0.004s
I20260928 08:31:44.407521 26543 heartbeater.cc:503] Master 127.15.134.254:40203 was elected leader, sending a full tablet report...
I20260928 08:31:44.407886 26699 log.cc:824] T 4f72955259d44739839853ae5b35b8df P 0c7501ed02814400b9108398fe6aee29: Log is configured to *not* fsync() on all Append() calls
I20260928 08:31:44.414542 26699 tablet_bootstrap.cc:492] T 4f72955259d44739839853ae5b35b8df P 0c7501ed02814400b9108398fe6aee29: No bootstrap required, opened a new log
I20260928 08:31:44.414830 26699 ts_tablet_manager.cc:1397] T 4f72955259d44739839853ae5b35b8df P 0c7501ed02814400b9108398fe6aee29: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260928 08:31:44.426661 26699 raft_consensus.cc:348] T 4f72955259d44739839853ae5b35b8df P 0c7501ed02814400b9108398fe6aee29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d275ba698146768378aa019bbe3930" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 41625 } } peers { permanent_uuid: "6092e2b6d0d64fa98e2f638fcc63b447" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 41421 } } peers { permanent_uuid: "0c7501ed02814400b9108398fe6aee29" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 43791 } }
I20260928 08:31:44.427227 26699 raft_consensus.cc:374] T 4f72955259d44739839853ae5b35b8df P 0c7501ed02814400b9108398fe6aee29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:31:44.427356 26699 raft_consensus.cc:729] T 4f72955259d44739839853ae5b35b8df P 0c7501ed02814400b9108398fe6aee29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c7501ed02814400b9108398fe6aee29, State: Initialized, Role: FOLLOWER
W20260928 08:31:44.428172 26410 tablet.cc:2369] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:31:44.428252 26699 consensus_queue.cc:260] T 4f72955259d44739839853ae5b35b8df P 0c7501ed02814400b9108398fe6aee29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d275ba698146768378aa019bbe3930" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 41625 } } peers { permanent_uuid: "6092e2b6d0d64fa98e2f638fcc63b447" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 41421 } } peers { permanent_uuid: "0c7501ed02814400b9108398fe6aee29" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 43791 } }
I20260928 08:31:44.430086 26676 heartbeater.cc:503] Master 127.15.134.254:40203 was elected leader, sending a full tablet report...
I20260928 08:31:44.430588 26699 ts_tablet_manager.cc:1428] T 4f72955259d44739839853ae5b35b8df P 0c7501ed02814400b9108398fe6aee29: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
W20260928 08:31:44.440498 26544 tablet.cc:2369] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:31:44.463356 26677 tablet.cc:2369] T 4f72955259d44739839853ae5b35b8df P 0c7501ed02814400b9108398fe6aee29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:31:44.492220 26700 raft_consensus.cc:482] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:31:44.492532 26700 raft_consensus.cc:504] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d275ba698146768378aa019bbe3930" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 41625 } } peers { permanent_uuid: "6092e2b6d0d64fa98e2f638fcc63b447" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 41421 } } peers { permanent_uuid: "0c7501ed02814400b9108398fe6aee29" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 43791 } }
I20260928 08:31:44.494583 26700 leader_election.cc:288] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6092e2b6d0d64fa98e2f638fcc63b447 (127.15.134.193:41421), 0c7501ed02814400b9108398fe6aee29 (127.15.134.195:43791)
I20260928 08:31:44.503996 26692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:44.495178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:36359 (local address 127.15.134.193:41421)
0928 08:31:44.495421 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:44.495428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:44.497411 (+ 1983us) server_negotiation.cc:407] Connection header received
0928 08:31:44.497757 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:44.497764 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:44.497847 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:44.497939 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:44.498614 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.499922 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.502274 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.503042 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.503225 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:44.503313 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:44.503621 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 08:31:44.503749 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 08:31:44.504145 26704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:44.495550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:41421 (local address 127.15.134.194:36359)
0928 08:31:44.497268 (+ 1718us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:44.497296 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:44.497566 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:44.497980 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:44.497984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:44.498014 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:44.498442 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:44.498453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.500069 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.500073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:44.502134 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:44.502152 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.503309 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.503315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:44.503465 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:44.503487 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 08:31:44.503593 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 08:31:44.503718 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1541,"mutex_wait_us":1,"thread_start_us":1286,"threads_started":1}
I20260928 08:31:44.504803 26701 raft_consensus.cc:482] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:31:44.505239 26693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:44.495169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:55013 (local address 127.15.134.195:43791)
0928 08:31:44.495370 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:44.495375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:44.497797 (+ 2422us) server_negotiation.cc:407] Connection header received
0928 08:31:44.497972 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:44.497980 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:44.498068 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:44.498167 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:44.498808 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.499577 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.502143 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.503781 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.503981 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:44.504083 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:44.504731 (+ 648us) server_negotiation.cc:299] Negotiation successful
0928 08:31:44.504909 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:31:44.505132 26701 raft_consensus.cc:504] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d275ba698146768378aa019bbe3930" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 41625 } } peers { permanent_uuid: "6092e2b6d0d64fa98e2f638fcc63b447" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 41421 } } peers { permanent_uuid: "0c7501ed02814400b9108398fe6aee29" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 43791 } }
I20260928 08:31:44.505424 26364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f72955259d44739839853ae5b35b8df" candidate_uuid: "14d275ba698146768378aa019bbe3930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6092e2b6d0d64fa98e2f638fcc63b447" is_pre_election: true
I20260928 08:31:44.505918 26364 raft_consensus.cc:2454] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d275ba698146768378aa019bbe3930 in term 0.
I20260928 08:31:44.506136 26705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:44.495616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:43791 (local address 127.15.134.194:55013)
0928 08:31:44.497672 (+ 2056us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:44.497692 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:44.497820 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:44.498207 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:44.498211 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:44.498231 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:44.498669 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:44.498677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.499713 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.499719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:44.501970 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:44.501985 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.504251 (+ 2266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.504259 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:44.504480 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:44.504530 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 08:31:44.505480 (+ 950us) client_negotiation.cc:240] Negotiation successful
0928 08:31:44.505656 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1919,"mutex_wait_us":25,"thread_start_us":1720,"threads_started":1}
I20260928 08:31:44.506699 26432 leader_election.cc:302] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d275ba698146768378aa019bbe3930, 6092e2b6d0d64fa98e2f638fcc63b447; no voters:
I20260928 08:31:44.507114 26701 leader_election.cc:288] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14d275ba698146768378aa019bbe3930 (127.15.134.194:41625), 0c7501ed02814400b9108398fe6aee29 (127.15.134.195:43791)
I20260928 08:31:44.507385 26700 raft_consensus.cc:2781] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:31:44.507522 26700 raft_consensus.cc:482] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:31:44.507637 26700 raft_consensus.cc:3037] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:44.507540 26631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f72955259d44739839853ae5b35b8df" candidate_uuid: "14d275ba698146768378aa019bbe3930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c7501ed02814400b9108398fe6aee29" is_pre_election: true
I20260928 08:31:44.508131 26631 raft_consensus.cc:2454] T 4f72955259d44739839853ae5b35b8df P 0c7501ed02814400b9108398fe6aee29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d275ba698146768378aa019bbe3930 in term 0.
I20260928 08:31:44.511336 26700 raft_consensus.cc:504] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d275ba698146768378aa019bbe3930" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 41625 } } peers { permanent_uuid: "6092e2b6d0d64fa98e2f638fcc63b447" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 41421 } } peers { permanent_uuid: "0c7501ed02814400b9108398fe6aee29" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 43791 } }
I20260928 08:31:44.512394 26364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f72955259d44739839853ae5b35b8df" candidate_uuid: "14d275ba698146768378aa019bbe3930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6092e2b6d0d64fa98e2f638fcc63b447"
I20260928 08:31:44.512573 26631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f72955259d44739839853ae5b35b8df" candidate_uuid: "14d275ba698146768378aa019bbe3930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c7501ed02814400b9108398fe6aee29"
I20260928 08:31:44.512637 26364 raft_consensus.cc:3037] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:44.512801 26631 raft_consensus.cc:3037] T 4f72955259d44739839853ae5b35b8df P 0c7501ed02814400b9108398fe6aee29 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:31:44.515952 26364 raft_consensus.cc:2454] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d275ba698146768378aa019bbe3930 in term 1.
I20260928 08:31:44.516366 26432 leader_election.cc:302] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d275ba698146768378aa019bbe3930, 6092e2b6d0d64fa98e2f638fcc63b447; no voters:
I20260928 08:31:44.516692 26631 raft_consensus.cc:2454] T 4f72955259d44739839853ae5b35b8df P 0c7501ed02814400b9108398fe6aee29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d275ba698146768378aa019bbe3930 in term 1.
I20260928 08:31:44.517537 26700 leader_election.cc:288] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930 [CANDIDATE]: Term 1 election: Requested vote from peers 6092e2b6d0d64fa98e2f638fcc63b447 (127.15.134.193:41421), 0c7501ed02814400b9108398fe6aee29 (127.15.134.195:43791)
I20260928 08:31:44.517794 26700 raft_consensus.cc:2781] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:31:44.517973 26700 raft_consensus.cc:686] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930 [term 1 LEADER]: Becoming Leader. State: Replica: 14d275ba698146768378aa019bbe3930, State: Running, Role: LEADER
I20260928 08:31:44.518656 26700 consensus_queue.cc:237] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d275ba698146768378aa019bbe3930" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 41625 } } peers { permanent_uuid: "6092e2b6d0d64fa98e2f638fcc63b447" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 41421 } } peers { permanent_uuid: "0c7501ed02814400b9108398fe6aee29" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 43791 } }
I20260928 08:31:44.531574 26690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:44.508004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:43271 (local address 127.15.134.194:41625)
0928 08:31:44.508200 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:44.508208 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:44.508982 (+ 774us) server_negotiation.cc:407] Connection header received
0928 08:31:44.509212 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:44.509218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:44.509282 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:44.509337 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:44.510057 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.511141 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.528323 (+ 17182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.529749 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.529879 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:44.529974 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:44.531145 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0928 08:31:44.531310 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 08:31:44.532451 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:44.507524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:41625 (local address 127.15.134.193:43271)
0928 08:31:44.508883 (+ 1359us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:44.508904 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:44.509078 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:44.509527 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:44.509531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:44.509557 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:44.509897 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:44.509907 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.511323 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.511330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:44.524965 (+ 13635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:44.524987 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.530227 (+ 5240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.530237 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:44.530417 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:44.530462 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 08:31:44.531904 (+ 1442us) client_negotiation.cc:240] Negotiation successful
0928 08:31:44.532055 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1231,"thread_start_us":1556,"threads_started":1}
I20260928 08:31:44.531708 26219 catalog_manager.cc:5495] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14d275ba698146768378aa019bbe3930 (127.15.134.194). New cstate: current_term: 1 leader_uuid: "14d275ba698146768378aa019bbe3930" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14d275ba698146768378aa019bbe3930" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 41625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6092e2b6d0d64fa98e2f638fcc63b447" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 41421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c7501ed02814400b9108398fe6aee29" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 43791 } health_report { overall_health: UNKNOWN } } }
I20260928 08:31:44.534232 26693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:44.510023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:43065 (local address 127.15.134.195:43791)
0928 08:31:44.510207 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:44.510214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:44.522533 (+ 12319us) server_negotiation.cc:407] Connection header received
0928 08:31:44.523043 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:44.523051 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:44.523140 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:44.523208 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:31:44.524212 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.525688 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.532173 (+ 6485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.533224 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.533322 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:44.533369 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:44.533808 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 08:31:44.533932 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 08:31:44.534256 26707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:44.509528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:43791 (local address 127.15.134.193:43065)
0928 08:31:44.522409 (+ 12881us) negotiation.cc:107] Waiting for socket to connect
0928 08:31:44.522432 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:31:44.522838 (+ 406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:31:44.523425 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:31:44.523431 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:31:44.523461 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:31:44.523990 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:44.524005 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.526011 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.526019 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:44.531989 (+ 5970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:31:44.532002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.533521 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.533526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:31:44.533639 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:44.533659 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 08:31:44.533772 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 08:31:44.533897 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12713,"thread_start_us":13016,"threads_started":1}
I20260928 08:31:44.535354 26631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f72955259d44739839853ae5b35b8df" candidate_uuid: "6092e2b6d0d64fa98e2f638fcc63b447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c7501ed02814400b9108398fe6aee29" is_pre_election: true
I20260928 08:31:44.535732 26631 raft_consensus.cc:2379] T 4f72955259d44739839853ae5b35b8df P 0c7501ed02814400b9108398fe6aee29 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6092e2b6d0d64fa98e2f638fcc63b447 in current term 1: Already voted for candidate 14d275ba698146768378aa019bbe3930 in this term.
I20260928 08:31:44.535585 26498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f72955259d44739839853ae5b35b8df" candidate_uuid: "6092e2b6d0d64fa98e2f638fcc63b447" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14d275ba698146768378aa019bbe3930" is_pre_election: true
I20260928 08:31:44.536978 26297 leader_election.cc:302] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6092e2b6d0d64fa98e2f638fcc63b447; no voters: 0c7501ed02814400b9108398fe6aee29, 14d275ba698146768378aa019bbe3930
I20260928 08:31:44.537443 26701 raft_consensus.cc:2726] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 08:31:44.577068 26690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:44.571030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55916 (local address 127.15.134.194:41625)
0928 08:31:44.571211 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:44.571217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:44.571330 (+ 113us) server_negotiation.cc:407] Connection header received
0928 08:31:44.571518 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:44.571524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:44.571624 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:44.571789 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:31:44.572145 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.573317 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.574708 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.575315 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.575416 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:44.575907 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:31:44.576458 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:31:44.576578 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:44.576761 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 08:31:44.576834 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 08:31:44.585865 26631 raft_consensus.cc:1261] T 4f72955259d44739839853ae5b35b8df P 0c7501ed02814400b9108398fe6aee29 [term 1 FOLLOWER]: Refusing update from remote peer 14d275ba698146768378aa019bbe3930: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:31:44.585981 26364 raft_consensus.cc:1261] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447 [term 1 FOLLOWER]: Refusing update from remote peer 14d275ba698146768378aa019bbe3930: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:31:44.586867 26708 consensus_queue.cc:1035] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c7501ed02814400b9108398fe6aee29" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 43791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:31:44.587229 26700 consensus_queue.cc:1035] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6092e2b6d0d64fa98e2f638fcc63b447" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 41421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:31:44.600471 26712 mvcc.cc:205] Tried to move back new op lower bound from 7334233311572496384 to 7334233311314472960. Current Snapshot: MvccSnapshot[applied={T|T < 7334233311572496384}]
I20260928 08:31:44.620134 26220 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:31:44.620457 26220 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:31:44.624579 26220 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33e693c83f814687928f0939290596e4: Sending DeleteTablet for 3 replicas of tablet 4f72955259d44739839853ae5b35b8df
I20260928 08:31:44.626282 26478 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f72955259d44739839853ae5b35b8df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:31:44 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:55908
I20260928 08:31:44.626497 26344 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f72955259d44739839853ae5b35b8df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:31:44 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:40978
I20260928 08:31:44.627421 26611 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f72955259d44739839853ae5b35b8df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:31:44 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:39144
I20260928 08:31:44.627635 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 26281
I20260928 08:31:44.631338 26722 tablet_replica.cc:330] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447: stopping tablet replica
I20260928 08:31:44.631944 26722 raft_consensus.cc:2229] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:31:44.632665 26722 raft_consensus.cc:2258] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:31:44.633668 26723 tablet_replica.cc:330] T 4f72955259d44739839853ae5b35b8df P 0c7501ed02814400b9108398fe6aee29: stopping tablet replica
I20260928 08:31:44.634249 26723 raft_consensus.cc:2229] T 4f72955259d44739839853ae5b35b8df P 0c7501ed02814400b9108398fe6aee29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:31:44.634877 26723 raft_consensus.cc:2258] T 4f72955259d44739839853ae5b35b8df P 0c7501ed02814400b9108398fe6aee29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:31:44.636662 26721 tablet_replica.cc:330] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930: stopping tablet replica
I20260928 08:31:44.637105 26721 raft_consensus.cc:2229] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:31:44.637683 26721 raft_consensus.cc:2258] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:31:44.639593 26722 ts_tablet_manager.cc:1905] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:31:44.639961 26721 ts_tablet_manager.cc:1905] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:31:44.646663 26723 ts_tablet_manager.cc:1905] T 4f72955259d44739839853ae5b35b8df P 0c7501ed02814400b9108398fe6aee29: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 08:31:44 dist-test-slave-qp2n krb5kdc[26175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.193@KRBTEST.COM
I20260928 08:31:44.647522 26721 ts_tablet_manager.cc:1918] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:31:44.647786 26721 log.cc:1196] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/4f72955259d44739839853ae5b35b8df
I20260928 08:31:44.648424 26721 ts_tablet_manager.cc:1939] T 4f72955259d44739839853ae5b35b8df P 14d275ba698146768378aa019bbe3930: Deleting consensus metadata
I20260928 08:31:44.650456 26204 catalog_manager.cc:4841] TS 14d275ba698146768378aa019bbe3930 (127.15.134.194:41625): tablet 4f72955259d44739839853ae5b35b8df (table test-table [id=e256afa161eb47bcbc69e18d1336ed6a]) successfully deleted
I20260928 08:31:44.653004 26723 ts_tablet_manager.cc:1918] T 4f72955259d44739839853ae5b35b8df P 0c7501ed02814400b9108398fe6aee29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:31:44.653182 26723 log.cc:1196] T 4f72955259d44739839853ae5b35b8df P 0c7501ed02814400b9108398fe6aee29: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/4f72955259d44739839853ae5b35b8df
I20260928 08:31:44.653699 26723 ts_tablet_manager.cc:1939] T 4f72955259d44739839853ae5b35b8df P 0c7501ed02814400b9108398fe6aee29: Deleting consensus metadata
I20260928 08:31:44.655666 26204 catalog_manager.cc:4841] TS 0c7501ed02814400b9108398fe6aee29 (127.15.134.195:43791): tablet 4f72955259d44739839853ae5b35b8df (table test-table [id=e256afa161eb47bcbc69e18d1336ed6a]) successfully deleted
I20260928 08:31:44.655921 26722 ts_tablet_manager.cc:1918] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:31:44.656190 26722 log.cc:1196] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790584205252592-15899-0/minicluster-data/ts-0/wal/wals/4f72955259d44739839853ae5b35b8df
I20260928 08:31:44.656764 26722 ts_tablet_manager.cc:1939] T 4f72955259d44739839853ae5b35b8df P 6092e2b6d0d64fa98e2f638fcc63b447: Deleting consensus metadata
I20260928 08:31:44.657630 26692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:44.628830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40988 (local address 127.15.134.193:41421)
0928 08:31:44.629016 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:44.629021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:44.636268 (+ 7247us) server_negotiation.cc:407] Connection header received
0928 08:31:44.636364 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:44.636370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:44.636456 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:44.636570 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:44.638612 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.639373 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.641574 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.642266 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.642375 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:44.648197 (+ 5822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:44.648215 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:44.648237 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:44.648297 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:44.652951 (+ 4654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:44.654160 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:44.654172 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:44.654176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:44.654376 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:44.654939 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:44.654945 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:44.654948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:44.655690 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:44.656028 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:44.656577 (+ 549us) server_negotiation.cc:299] Negotiation successful
0928 08:31:44.656977 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 08:31:44.658349 26404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:44.658751 26205 catalog_manager.cc:4841] TS 6092e2b6d0d64fa98e2f638fcc63b447 (127.15.134.193:41421): tablet 4f72955259d44739839853ae5b35b8df (table test-table [id=e256afa161eb47bcbc69e18d1336ed6a]) successfully deleted
I20260928 08:31:44.772943 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 26281
I20260928 08:31:44.792380 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 26415
Sep 28 08:31:44 dist-test-slave-qp2n krb5kdc[26175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.194@KRBTEST.COM
I20260928 08:31:44.810335 26690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:44.793270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55922 (local address 127.15.134.194:41625)
0928 08:31:44.793510 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:44.793516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:44.793538 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:31:44.793730 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:44.793736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:44.793819 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:44.793945 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:44.795191 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.796442 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.797860 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.798573 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.798704 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:44.803104 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:44.803120 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:44.803142 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:44.803197 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:44.807406 (+ 4209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:44.808085 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:44.808095 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:44.808099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:44.808258 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:44.808684 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:44.808689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:44.808693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:44.809196 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:44.809518 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:44.809733 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 08:31:44.809971 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:31:44.811107 26538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:44.926721 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 26415
I20260928 08:31:44.946070 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 26548
Sep 28 08:31:44 dist-test-slave-qp2n krb5kdc[26175](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.195@KRBTEST.COM
I20260928 08:31:44.962175 26693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:31:44.946825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39158 (local address 127.15.134.195:43791)
0928 08:31:44.946991 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 08:31:44.946996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:31:44.947014 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:31:44.947181 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:31:44.947184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:31:44.947260 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:31:44.947339 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:31:44.948884 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.949549 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.950986 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:31:44.951398 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:31:44.951520 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:31:44.956398 (+ 4878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:31:44.956415 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:31:44.956446 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:31:44.956490 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:31:44.959467 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:44.960170 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:44.960177 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:44.960180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:44.960308 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:31:44.960786 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:31:44.960790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:31:44.960792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:31:44.961156 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:31:44.961382 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 08:31:44.961659 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 08:31:44.961820 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 08:31:44.962814 26671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:45.054013 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 26548
I20260928 08:31:45.071965 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 26188
I20260928 08:31:45.073191 26250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:31:45.182232 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 26188
2026-09-28T08:31:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4436 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:31:45 dist-test-slave-qp2n krb5kdc[26731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:31:45 dist-test-slave-qp2n krb5kdc[26731](info): set up 2 sockets
Sep 28 08:31:45 dist-test-slave-qp2n krb5kdc[26731](info): commencing operation
krb5kdc: starting...
W20260928 08:31:46.299129 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:31:46 dist-test-slave-qp2n krb5kdc[26731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:31:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:31:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.134.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.134.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.134.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/rangeradmin_127.15.134.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.134.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/rangeradmin_127.15.134.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.134.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.134.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.134.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/rangerlookup_127.15.134.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.134.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/rangerlookup_127.15.134.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.134.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.134.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.134.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/HTTP_127.15.134.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.134.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/HTTP_127.15.134.212@KRBTEST.COM.keytab.
I20260928 08:31:46.640017 15899 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task0uZKZe/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/postgres -l logfile start
2026-09-28 08:31:49.385 UTC [26761] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 08:31:49.385 UTC [26761] LOG: listening on IPv4 address "127.15.134.212", port 49689
2026-09-28 08:31:49.389 UTC [26761] LOG: listening on Unix socket "/tmp/.s.PGSQL.49689"
2026-09-28 08:31:49.413 UTC [26764] LOG: database system was shut down at 2026-09-28 08:31:47 UTC
2026-09-28 08:31:49.417 UTC [26761] LOG: database system is ready to accept connections
I20260928 08:31:50.411322 15899 mini_postgres.cc:96] Postgres bound to 49689
2026-09-28 08:31:50.436 UTC [26772] FATAL: database "slave" does not exist
127.15.134.212:49689 - accepting connections
I20260928 08:31:50.437490 15899 mini_ranger.cc:161] Starting Ranger...
I20260928 08:31:50.466740 15899 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 08:31:51.022680 15899 mini_ranger.cc:87] Created ranger Postgres database
I20260928 08:31:51.022858 15899 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ranger-admin
2026-09-28 08:31:51,561 [I] DB FLAVOR :POSTGRES
2026-09-28 08:31:51,561 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 08:31:51,562 [I] Checking connection..
2026-09-28 08:31:51,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:49689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 08:31:51,839 [I] Checking connection passed.
2026-09-28 08:31:51,839 [I] --------- Verifying version history table ---------
2026-09-28 08:31:51,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:49689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 08:31:52,102 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 08:31:52,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:49689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 08:31:52,349 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 08:31:52,349 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 08:31:52,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:49689/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 08:31:52.596 UTC [26836] WARNING: there is no transaction in progress
2026-09-28 08:31:52,603 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 08:31:52,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:49689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 08:31:52,889 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 08:31:52,889 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 08:31:52,889 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:49689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 08:31:53,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:49689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 08:31:53,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:49689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-qp2n.c.gcp-upstream.internal','N') ;"
2026-09-28 08:31:53,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:49689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 08:31:53,905 [I] Table x_portal_user does not exist in database ranger
2026-09-28 08:31:53,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:49689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 08:31:54,150 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 08:31:54,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:49689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 08:31:54,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:49689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 08:31:54,645 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 08:31:54,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:49689/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 08:31:55.704 UTC [26944] WARNING: there is no transaction in progress
2026-09-28 08:31:55.723 UTC [26944] WARNING: there is no transaction in progress
2026-09-28 08:31:55.740 UTC [26944] WARNING: there is no transaction in progress
2026-09-28 08:31:55.757 UTC [26944] WARNING: there is no transaction in progress
2026-09-28 08:31:55.775 UTC [26944] WARNING: there is no transaction in progress
2026-09-28 08:31:55.796 UTC [26944] WARNING: there is no transaction in progress
2026-09-28 08:31:55.860 UTC [26944] WARNING: there is no transaction in progress
2026-09-28 08:31:55.868 UTC [26944] WARNING: there is no transaction in progress
2026-09-28 08:31:55.884 UTC [26944] WARNING: there is no transaction in progress
2026-09-28 08:31:55.896 UTC [26944] WARNING: there is no transaction in progress
2026-09-28 08:31:55.907 UTC [26944] WARNING: there is no transaction in progress
2026-09-28 08:31:55.922 UTC [26944] WARNING: there is no transaction in progress
2026-09-28 08:31:55.932 UTC [26944] WARNING: there is no transaction in progress
2026-09-28 08:31:55.940 UTC [26944] WARNING: there is no transaction in progress
2026-09-28 08:31:55.947 UTC [26944] WARNING: there is no transaction in progress
2026-09-28 08:31:55.955 UTC [26944] WARNING: there is no transaction in progress
2026-09-28 08:31:56,669 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 08:31:56,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:49689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 08:31:56,919 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 08:31:56,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:49689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 08:31:57,186 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 08:31:57,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:49689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 08:31:57,426 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 08:31:57,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:49689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 08:31:57,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:49689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 08:31:57,935 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:49689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-qp2n.c.gcp-upstream.internal';"
2026-09-28 08:31:58,194 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 08:31:58,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:49689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 08:31:58,432 [I] DB_PATCHES have already been applied
I20260928 08:31:58.439249 15899 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ranger-admin:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task0uZKZe/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 08:31:58.439350 15899 mini_ranger.cc:193] Using host: 127.15.134.212
I20260928 08:31:58.456100 15899 mini_ranger.cc:227] Ranger admin URL: http://127.15.134.212:40789
Sep 28, 2026 8:31:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 8:31:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 8:31:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 8:31:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 8:31:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 8:31:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 8:31:59 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 08:32:00.811193 15899 mini_ranger.cc:160] Time spent starting Ranger: real 10.374s user 0.000s sys 0.051s
Sep 28, 2026 8:32:22 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 8:32:22 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 8:32:22 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 8:32:22 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 8:32:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 8:32:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 8:32:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 8:32:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 8:32:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 8:32:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 8:32:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 8:32:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 8:32:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 8:32:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 8:32:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 8:32:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 8:32:22 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 8:32:23 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 08:32:24.271009 15899 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.134.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.134.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.134.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/rangerkms_127.15.134.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.134.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/rangerkms_127.15.134.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.134.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.134.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.134.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/HTTP_127.15.134.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.134.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/HTTP_127.15.134.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 08:32:24 dist-test-slave-qp2n krb5kdc[26731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584344, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 08:32:24.471408 15899 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 08:32:24.494696 15899 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 08:32:25.469054 15899 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 08:32:25.469249 15899 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ranger-kms
I20260928 08:32:25.469396 15899 mini_ranger_kms.cc:227] Using postgres at 127.15.134.212:49689
2026-09-28 08:32:26,359 [I] DB FLAVOR :POSTGRES
2026-09-28 08:32:26,360 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 08:32:26,360 [I] Checking connection
2026-09-28 08:32:26,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:49689/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 08:32:26,601 [I] connection success
2026-09-28 08:32:26,602 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 08:32:26,602 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 08:32:26,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:49689/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 08:32:26,869 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 08:32:26,869 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 08:32:26,870 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 08:32:26,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:49689/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 08:32:27,169 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 08:32:27.659652 15899 mini_ranger_kms.cc:310] Created kms service
I20260928 08:32:27.867014 15899 mini_ranger_kms.cc:326] Created kudu user
I20260928 08:32:27.916055 15899 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 08:32:27 dist-test-slave-qp2n krb5kdc[26731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584344, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.134.212@KRBTEST.COM
I20260928 08:32:28.330396 15899 mini_ranger_kms.cc:384] Added ranger policy
I20260928 08:32:28.330711 15899 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ranger-kms:/tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 08:32:28.330780 15899 mini_ranger_kms.cc:240] Using host: 127.15.134.212
I20260928 08:32:28.354139 15899 mini_ranger_kms.cc:276] Ranger KMS PID: 27126
I20260928 08:32:28.354347 15899 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.134.212:35639
Sep 28, 2026 8:32:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 8:32:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 8:32:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 8:32:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 8:32:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 8:32:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 08:32:30.649242 15899 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.178s user 0.139s sys 0.417s
I20260928 08:32:30.649502 15899 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 08:32:30.649718 15899 mini_ranger_kms.cc:401] 127.15.134.212:35639/kms/v1/keys
Sep 28 08:32:30 dist-test-slave-qp2n krb5kdc[26731](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790584350, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.134.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 08:32:31 dist-test-slave-qp2n krb5kdc[26731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584350, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.134.212@KRBTEST.COM for HTTP/127.15.134.212@KRBTEST.COM
Sep 28 08:32:31 dist-test-slave-qp2n krb5kdc[26731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584350, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.134.212@KRBTEST.COM for HTTP/127.15.134.212@KRBTEST.COM
Sep 28 08:32:34 dist-test-slave-qp2n krb5kdc[26731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:32:34.136721 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:37799
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:37799
--ranger_config_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.134.212:35639/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:32:34.556337 27176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:32:34.556746 27176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:32:34.556857 27176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:32:34.565683 27176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:32:34.565806 27176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:32:34.565872 27176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:32:34.565961 27176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:32:34.566026 27176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:32:34.577932 27176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.134.212:35639/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:37799
--ranger_config_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:37799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:32:34.579222 27176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:32:34.580497 27176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:32:34.594261 27181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:32:34.594511 27176 server_base.cc:1034] running on GCE node
W20260928 08:32:34.594287 27184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:32:34.594219 27182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:32:34.595547 27176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:32:34.597146 27176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:32:34.598390 27176 hybrid_clock.cc:648] HybridClock initialized: now 1790584354598325 us; error 48 us; skew 500 ppm
I20260928 08:32:34.598765 27176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:32:34 dist-test-slave-qp2n krb5kdc[26731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584354, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:32:34.602238 27176 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:32:34.604169 27176 webserver.cc:460] Webserver started at http://127.15.134.254:38177/ using document root <none> and password file <none>
I20260928 08:32:34.604982 27176 fs_manager.cc:359] Metadata directory not provided
I20260928 08:32:34.605141 27176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:32:34.605490 27176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:32:34 dist-test-slave-qp2n krb5kdc[26731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584354, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for HTTP/127.15.134.212@KRBTEST.COM
I20260928 08:32:34.668157 27176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "a2867122358a45e9ac6c47c86659919c"
format_stamp: "Formatted at 2026-09-28 08:32:34 on dist-test-slave-qp2n"
server_key: "45fb59bb96212ff9508de4ff96297bcc"
server_key_iv: "c46bb78da45ebbc64c5f58e299fdb753"
server_key_version: "kuduclusterkey@0"
I20260928 08:32:34.669260 27176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "a2867122358a45e9ac6c47c86659919c"
format_stamp: "Formatted at 2026-09-28 08:32:34 on dist-test-slave-qp2n"
server_key: "45fb59bb96212ff9508de4ff96297bcc"
server_key_iv: "c46bb78da45ebbc64c5f58e299fdb753"
server_key_version: "kuduclusterkey@0"
I20260928 08:32:34.675722 27176 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 08:32:34.697281 27191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:32:34.698925 27176 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 08:32:34.699133 27176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "a2867122358a45e9ac6c47c86659919c"
format_stamp: "Formatted at 2026-09-28 08:32:34 on dist-test-slave-qp2n"
server_key: "45fb59bb96212ff9508de4ff96297bcc"
server_key_iv: "c46bb78da45ebbc64c5f58e299fdb753"
server_key_version: "kuduclusterkey@0"
I20260928 08:32:34.699368 27176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:32:34.723209 27176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:32:34.728029 27176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:32:34.728468 27176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:32:34.750602 27176 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:37799
I20260928 08:32:34.750619 27242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:37799 every 8 connection(s)
I20260928 08:32:34.752096 27176 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:32:34.756179 27243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:32:34.756790 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 27176
I20260928 08:32:34.756999 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:32:34.757661 15899 external_mini_cluster.cc:1428] Setting key 6fd17391bc0b05d37aa7ced5bc0351e6
I20260928 08:32:34.771142 27243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2867122358a45e9ac6c47c86659919c: Bootstrap starting.
Sep 28 08:32:34 dist-test-slave-qp2n krb5kdc[26731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:32:34.776098 27243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2867122358a45e9ac6c47c86659919c: Neither blocks nor log segments found. Creating new log.
I20260928 08:32:34.778391 27243 log.cc:824] T 00000000000000000000000000000000 P a2867122358a45e9ac6c47c86659919c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:32:34.781612 27243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2867122358a45e9ac6c47c86659919c: No bootstrap required, opened a new log
I20260928 08:32:34.784446 27246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:32:34.759716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44084 (local address 127.15.134.254:37799)
0928 08:32:34.761126 (+ 1410us) server_negotiation.cc:206] Beginning negotiation
0928 08:32:34.761142 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 08:32:34.761188 (+ 46us) server_negotiation.cc:407] Connection header received
0928 08:32:34.763567 (+ 2379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:32:34.763600 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:32:34.764111 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:32:34.764593 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:32:34.766132 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:34.768290 (+ 2158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:34.769849 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:34.770617 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:34.770856 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:32:34.775538 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:32:34.775550 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:32:34.775570 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:32:34.775624 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:32:34.779744 (+ 4120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:32:34.780573 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:32:34.780582 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:32:34.780589 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:32:34.780832 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:32:34.781311 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:32:34.781316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:32:34.781318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:32:34.781940 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:32:34.782316 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0928 08:32:34.782818 (+ 502us) server_negotiation.cc:299] Negotiation successful
0928 08:32:34.783297 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1166,"thread_start_us":793,"threads_started":1}
I20260928 08:32:34.790653 27243 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2867122358a45e9ac6c47c86659919c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2867122358a45e9ac6c47c86659919c" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 37799 } }
I20260928 08:32:34.791074 27243 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2867122358a45e9ac6c47c86659919c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:32:34.791188 27243 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2867122358a45e9ac6c47c86659919c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2867122358a45e9ac6c47c86659919c, State: Initialized, Role: FOLLOWER
I20260928 08:32:34.791885 27243 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2867122358a45e9ac6c47c86659919c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2867122358a45e9ac6c47c86659919c" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 37799 } }
I20260928 08:32:34.792096 27243 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2867122358a45e9ac6c47c86659919c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:32:34.792196 27243 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2867122358a45e9ac6c47c86659919c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:32:34.792311 27243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2867122358a45e9ac6c47c86659919c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:32:34.794450 27243 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2867122358a45e9ac6c47c86659919c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2867122358a45e9ac6c47c86659919c" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 37799 } }
I20260928 08:32:34.794936 27243 leader_election.cc:302] T 00000000000000000000000000000000 P a2867122358a45e9ac6c47c86659919c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2867122358a45e9ac6c47c86659919c; no voters:
I20260928 08:32:34.795544 27243 leader_election.cc:288] T 00000000000000000000000000000000 P a2867122358a45e9ac6c47c86659919c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:32:34.795691 27248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2867122358a45e9ac6c47c86659919c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:32:34.796968 27248 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2867122358a45e9ac6c47c86659919c [term 1 LEADER]: Becoming Leader. State: Replica: a2867122358a45e9ac6c47c86659919c, State: Running, Role: LEADER
I20260928 08:32:34.797498 27248 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2867122358a45e9ac6c47c86659919c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2867122358a45e9ac6c47c86659919c" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 37799 } }
I20260928 08:32:34.798032 27243 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2867122358a45e9ac6c47c86659919c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:32:34.801668 27249 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2867122358a45e9ac6c47c86659919c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2867122358a45e9ac6c47c86659919c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2867122358a45e9ac6c47c86659919c" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 37799 } } }
I20260928 08:32:34.802093 27249 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2867122358a45e9ac6c47c86659919c [sys.catalog]: This master's current role is: LEADER
I20260928 08:32:34.801663 27250 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2867122358a45e9ac6c47c86659919c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2867122358a45e9ac6c47c86659919c. Latest consensus state: current_term: 1 leader_uuid: "a2867122358a45e9ac6c47c86659919c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2867122358a45e9ac6c47c86659919c" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 37799 } } }
I20260928 08:32:34.802270 27250 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2867122358a45e9ac6c47c86659919c [sys.catalog]: This master's current role is: LEADER
I20260928 08:32:34.803814 27254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:32:34.809082 27254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:32:34.813640 27243 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 08:32:34.842299 27254 catalog_manager.cc:1347] Generated new cluster ID: fa87da81f0174335be790242a21f4eac
I20260928 08:32:34.842453 27254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:32:34.866549 27254 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:32:34.869088 27254 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:32:34.881846 27254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2867122358a45e9ac6c47c86659919c: Generated new TSK 0
I20260928 08:32:34.883229 27254 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.134.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
I20260928 08:32:35.696878 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.134.212:35639/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:0
--local_ip_for_outbound_sockets=127.15.134.193
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:37799
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:32:35 dist-test-slave-qp2n krb5kdc[26731](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790584355, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 08:32:36.184559 27288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:32:36.185029 27288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:32:36.185231 27288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:32:36.199425 27288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:32:36.199616 27288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:32:36.199841 27288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:32:36.219384 27288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.134.212:35639/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:37799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:32:36.221310 27288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:32:36.223075 27288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:32:36.260947 27299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:32:36.264369 27301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:32:36.264914 27288 server_base.cc:1034] running on GCE node
W20260928 08:32:36.265051 27298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:32:36.266170 27288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:32:36.267627 27288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:32:36.269040 27288 hybrid_clock.cc:648] HybridClock initialized: now 1790584356268874 us; error 51 us; skew 500 ppm
I20260928 08:32:36.269649 27288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:32:36 dist-test-slave-qp2n krb5kdc[26731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584356, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:32:36.277761 27288 init.cc:377] Logged in from keytab as kudu/127.15.134.193@KRBTEST.COM (short username kudu)
I20260928 08:32:36.279636 27288 webserver.cc:460] Webserver started at http://127.15.134.193:46353/ using document root <none> and password file <none>
I20260928 08:32:36.280469 27288 fs_manager.cc:359] Metadata directory not provided
I20260928 08:32:36.280689 27288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:32:36.281050 27288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:32:36 dist-test-slave-qp2n krb5kdc[26731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584356, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for HTTP/127.15.134.212@KRBTEST.COM
I20260928 08:32:36.301054 27288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-0/data/instance:
uuid: "817100cfa3b9403398bfb15320a0db35"
format_stamp: "Formatted at 2026-09-28 08:32:36 on dist-test-slave-qp2n"
server_key: "b9e4f2bc33a00af234589018123063c3"
server_key_iv: "c9b80bf85ee9136aafac0d4522a331ef"
server_key_version: "kuduclusterkey@0"
I20260928 08:32:36.302109 27288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance:
uuid: "817100cfa3b9403398bfb15320a0db35"
format_stamp: "Formatted at 2026-09-28 08:32:36 on dist-test-slave-qp2n"
server_key: "b9e4f2bc33a00af234589018123063c3"
server_key_iv: "c9b80bf85ee9136aafac0d4522a331ef"
server_key_version: "kuduclusterkey@0"
I20260928 08:32:36.313208 27288 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.006s sys 0.005s
I20260928 08:32:36.359721 27308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:32:36.362277 27288 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.008s sys 0.012s
I20260928 08:32:36.362654 27288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "817100cfa3b9403398bfb15320a0db35"
format_stamp: "Formatted at 2026-09-28 08:32:36 on dist-test-slave-qp2n"
server_key: "b9e4f2bc33a00af234589018123063c3"
server_key_iv: "c9b80bf85ee9136aafac0d4522a331ef"
server_key_version: "kuduclusterkey@0"
I20260928 08:32:36.363059 27288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:32:36.410898 27288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:32:36.422060 27288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:32:36.422703 27288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:32:36.429602 27288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:32:36.433161 27288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:32:36.433269 27288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:32:36.433404 27288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:32:36.433468 27288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:32:36 dist-test-slave-qp2n krb5kdc[26731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584355, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for HTTP/127.15.134.212@KRBTEST.COM
I20260928 08:32:36.687734 27288 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:41859
I20260928 08:32:36.687750 27420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:41859 every 8 connection(s)
I20260928 08:32:36.689280 27288 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:32:36.696309 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 27288
I20260928 08:32:36.696516 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance
I20260928 08:32:36.697361 15899 external_mini_cluster.cc:1428] Setting key 93ced896198a20d81e72ba32381a49e9
Sep 28 08:32:36 dist-test-slave-qp2n krb5kdc[26731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584356, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:32:36.718479 27423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:32:36.693073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:49945 (local address 127.15.134.254:37799)
0928 08:32:36.695149 (+ 2076us) server_negotiation.cc:206] Beginning negotiation
0928 08:32:36.695159 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:32:36.698576 (+ 3417us) server_negotiation.cc:407] Connection header received
0928 08:32:36.701707 (+ 3131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:32:36.701712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:32:36.701815 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:32:36.701939 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:32:36.704095 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:36.705076 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:36.706615 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:36.707030 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:36.707160 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:32:36.711330 (+ 4170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:32:36.711341 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:32:36.711344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:32:36.711376 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:32:36.714325 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:32:36.715358 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:32:36.715366 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:32:36.715373 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:32:36.715534 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:32:36.716083 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:32:36.716090 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:32:36.716096 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:32:36.716464 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:32:36.716588 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 08:32:36.717687 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0928 08:32:36.717952 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1903,"thread_start_us":1846,"threads_started":1}
I20260928 08:32:36.721061 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:32:36.694603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:37799 (local address 127.15.134.193:49945)
0928 08:32:36.697432 (+ 2829us) negotiation.cc:107] Waiting for socket to connect
0928 08:32:36.697490 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 08:32:36.701438 (+ 3948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:32:36.702234 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:32:36.702243 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:32:36.702999 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:32:36.703916 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:32:36.703948 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:32:36.705226 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:32:36.705231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:32:36.706471 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:32:36.706483 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:32:36.707247 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:32:36.707255 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:32:36.707488 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:32:36.708181 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:32:36.708203 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:32:36.711105 (+ 2902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:32:36.714476 (+ 3371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:32:36.714488 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:32:36.714533 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:32:36.715181 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:32:36.715685 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:32:36.715692 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:32:36.715698 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:32:36.715939 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:32:36.716731 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:32:36.716742 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:32:36.717246 (+ 504us) client_negotiation.cc:769] Sending connection context
0928 08:32:36.718787 (+ 1541us) client_negotiation.cc:240] Negotiation successful
0928 08:32:36.719451 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2529,"thread_start_us":2283,"threads_started":1}
I20260928 08:32:36.723287 27421 heartbeater.cc:347] Connected to a master server at 127.15.134.254:37799
I20260928 08:32:36.723632 27421 heartbeater.cc:464] Registering TS with master...
I20260928 08:32:36.724424 27421 heartbeater.cc:511] Master 127.15.134.254:37799 requested a full tablet report, sending...
I20260928 08:32:36.726863 27208 ts_manager.cc:194] Registered new tserver with Master: 817100cfa3b9403398bfb15320a0db35 (127.15.134.193:41859)
I20260928 08:32:36.728310 27208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.193@KRBTEST.COM'} at 127.15.134.193:49945
WARNING: no policy specified for kudu/127.15.134.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
I20260928 08:32:36.768821 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.134.212:35639/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:0
--local_ip_for_outbound_sockets=127.15.134.194
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:37799
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:32:36 dist-test-slave-qp2n krb5kdc[26731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584355, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for HTTP/127.15.134.212@KRBTEST.COM
W20260928 08:32:37.243738 27427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:32:37.244019 27427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:32:37.244153 27427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:32:37.251578 27427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:32:37.251694 27427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:32:37.251910 27427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:32:37.262079 27427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.134.212:35639/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:37799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:32:37.263478 27427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:32:37.264847 27427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:32:37.279865 27442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:32:37.280886 27427 server_base.cc:1034] running on GCE node
W20260928 08:32:37.283856 27441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:32:37.284235 27444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:32:37.285049 27427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:32:37.286075 27427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:32:37.287328 27427 hybrid_clock.cc:648] HybridClock initialized: now 1790584357287273 us; error 54 us; skew 500 ppm
I20260928 08:32:37.287689 27427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:32:37 dist-test-slave-qp2n krb5kdc[26731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584357, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:32:37.291163 27427 init.cc:377] Logged in from keytab as kudu/127.15.134.194@KRBTEST.COM (short username kudu)
I20260928 08:32:37.292999 27427 webserver.cc:460] Webserver started at http://127.15.134.194:45717/ using document root <none> and password file <none>
I20260928 08:32:37.293785 27427 fs_manager.cc:359] Metadata directory not provided
I20260928 08:32:37.293938 27427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:32:37.294317 27427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:32:37 dist-test-slave-qp2n krb5kdc[26731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584357, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for HTTP/127.15.134.212@KRBTEST.COM
I20260928 08:32:37.313602 27427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-1/data/instance:
uuid: "f74a37b2b4494135a206d5604c9ede6f"
format_stamp: "Formatted at 2026-09-28 08:32:37 on dist-test-slave-qp2n"
server_key: "7091b9ff1b5868e4248b327f4352df55"
server_key_iv: "55b0780428bcd2227a35b03a0a88aea6"
server_key_version: "kuduclusterkey@0"
I20260928 08:32:37.314527 27427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance:
uuid: "f74a37b2b4494135a206d5604c9ede6f"
format_stamp: "Formatted at 2026-09-28 08:32:37 on dist-test-slave-qp2n"
server_key: "7091b9ff1b5868e4248b327f4352df55"
server_key_iv: "55b0780428bcd2227a35b03a0a88aea6"
server_key_version: "kuduclusterkey@0"
I20260928 08:32:37.320266 27427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 08:32:37.348754 27451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:32:37.350941 27427 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260928 08:32:37.351182 27427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "f74a37b2b4494135a206d5604c9ede6f"
format_stamp: "Formatted at 2026-09-28 08:32:37 on dist-test-slave-qp2n"
server_key: "7091b9ff1b5868e4248b327f4352df55"
server_key_iv: "55b0780428bcd2227a35b03a0a88aea6"
server_key_version: "kuduclusterkey@0"
I20260928 08:32:37.351472 27427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:32:37.384864 27427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:32:37.390998 27427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:32:37.391368 27427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:32:37.392647 27427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:32:37.394577 27427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:32:37.394670 27427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:32:37.394781 27427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:32:37.394840 27427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:32:37.451711 27427 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:44807
I20260928 08:32:37.451716 27563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:44807 every 8 connection(s)
I20260928 08:32:37.453843 27427 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:32:37.458559 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 27427
I20260928 08:32:37.458758 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance
I20260928 08:32:37.459250 15899 external_mini_cluster.cc:1428] Setting key 5abb93d5317242ce0ea118556978f57f
Sep 28 08:32:37 dist-test-slave-qp2n krb5kdc[26731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584357, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:32:37.484248 27566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:32:37.457311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:48855 (local address 127.15.134.254:37799)
0928 08:32:37.458154 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0928 08:32:37.458160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:32:37.459156 (+ 996us) server_negotiation.cc:407] Connection header received
0928 08:32:37.461828 (+ 2672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:32:37.461835 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:32:37.461938 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:32:37.462052 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:32:37.464312 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:37.465721 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:37.468696 (+ 2975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:37.469438 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:37.469599 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:32:37.474877 (+ 5278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:32:37.474886 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:32:37.474890 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:32:37.474969 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:32:37.479512 (+ 4543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:32:37.480793 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:32:37.480800 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:32:37.480803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:32:37.480999 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:32:37.481772 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:32:37.481777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:32:37.481780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:32:37.482221 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:32:37.482379 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 08:32:37.483537 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0928 08:32:37.483774 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":542,"threads_started":1}
WARNING: no policy specified for kudu/127.15.134.195@KRBTEST.COM; defaulting to no policy
I20260928 08:32:37.485526 27567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:32:37.457892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:37799 (local address 127.15.134.194:48855)
0928 08:32:37.458968 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0928 08:32:37.459023 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 08:32:37.461514 (+ 2491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:32:37.462327 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:32:37.462340 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:32:37.463253 (+ 913us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:32:37.464108 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:32:37.464135 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:32:37.465959 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:32:37.465964 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:32:37.468504 (+ 2540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:32:37.468520 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:32:37.469707 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:32:37.469716 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:32:37.470032 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:32:37.470840 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:32:37.470874 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:32:37.474557 (+ 3683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:32:37.479713 (+ 5156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:32:37.479725 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:32:37.479746 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:32:37.480645 (+ 899us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:32:37.481135 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:32:37.481143 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:32:37.481149 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:32:37.481612 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:32:37.482377 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:32:37.482387 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:32:37.482917 (+ 530us) client_negotiation.cc:769] Sending connection context
0928 08:32:37.483341 (+ 424us) client_negotiation.cc:240] Negotiation successful
0928 08:32:37.483916 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":1770,"threads_started":1}
I20260928 08:32:37.487324 27564 heartbeater.cc:347] Connected to a master server at 127.15.134.254:37799
I20260928 08:32:37.487740 27564 heartbeater.cc:464] Registering TS with master...
I20260928 08:32:37.488683 27564 heartbeater.cc:511] Master 127.15.134.254:37799 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.195@KRBTEST.COM" created.
I20260928 08:32:37.490540 27208 ts_manager.cc:194] Registered new tserver with Master: f74a37b2b4494135a206d5604c9ede6f (127.15.134.194:44807)
I20260928 08:32:37.491838 27208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.194@KRBTEST.COM'} at 127.15.134.194:48855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
I20260928 08:32:37.520032 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.134.212:35639/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:0
--local_ip_for_outbound_sockets=127.15.134.195
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:37799
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:32:37.732547 27421 heartbeater.cc:503] Master 127.15.134.254:37799 was elected leader, sending a full tablet report...
W20260928 08:32:37.928005 27570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:32:37.928400 27570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:32:37.928656 27570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:32:37.937371 27570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:32:37.937479 27570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:32:37.937680 27570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:32:37.948206 27570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.134.212:35639/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:37799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:32:37.949815 27570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:32:37.951289 27570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:32:37.965847 27576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:32:37.965900 27575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:32:37.965955 27578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:32:37.972282 27570 server_base.cc:1034] running on GCE node
I20260928 08:32:37.973089 27570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:32:37.974090 27570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:32:37.975307 27570 hybrid_clock.cc:648] HybridClock initialized: now 1790584357975259 us; error 39 us; skew 500 ppm
I20260928 08:32:37.975659 27570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:32:37 dist-test-slave-qp2n krb5kdc[26731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584357, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:32:37.982625 27570 init.cc:377] Logged in from keytab as kudu/127.15.134.195@KRBTEST.COM (short username kudu)
I20260928 08:32:37.984537 27570 webserver.cc:460] Webserver started at http://127.15.134.195:45079/ using document root <none> and password file <none>
I20260928 08:32:37.985492 27570 fs_manager.cc:359] Metadata directory not provided
I20260928 08:32:37.985641 27570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:32:37.985985 27570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:32:37 dist-test-slave-qp2n krb5kdc[26731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584357, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for HTTP/127.15.134.212@KRBTEST.COM
I20260928 08:32:38.003608 27570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-2/data/instance:
uuid: "9dccaa9db2254fe59bfbf4b54a6519fe"
format_stamp: "Formatted at 2026-09-28 08:32:37 on dist-test-slave-qp2n"
server_key: "cfbfa0d4e3e2cfa530aabcfb7e7f8c7f"
server_key_iv: "eb71a37db57176ce5628d1adc8535c7a"
server_key_version: "kuduclusterkey@0"
I20260928 08:32:38.004540 27570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance:
uuid: "9dccaa9db2254fe59bfbf4b54a6519fe"
format_stamp: "Formatted at 2026-09-28 08:32:37 on dist-test-slave-qp2n"
server_key: "cfbfa0d4e3e2cfa530aabcfb7e7f8c7f"
server_key_iv: "eb71a37db57176ce5628d1adc8535c7a"
server_key_version: "kuduclusterkey@0"
I20260928 08:32:38.011479 27570 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 08:32:38.029723 27585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:32:38.031953 27570 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:32:38.032187 27570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "9dccaa9db2254fe59bfbf4b54a6519fe"
format_stamp: "Formatted at 2026-09-28 08:32:37 on dist-test-slave-qp2n"
server_key: "cfbfa0d4e3e2cfa530aabcfb7e7f8c7f"
server_key_iv: "eb71a37db57176ce5628d1adc8535c7a"
server_key_version: "kuduclusterkey@0"
I20260928 08:32:38.032473 27570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:32:38.048677 27570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:32:38.053210 27570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:32:38.053527 27570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:32:38.054495 27570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:32:38.056118 27570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:32:38.056219 27570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:32:38.056318 27570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:32:38.056362 27570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:32:38.098404 27570 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:38195
I20260928 08:32:38.098438 27697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:38195 every 8 connection(s)
I20260928 08:32:38.099751 27570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:32:38.101686 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 27570
I20260928 08:32:38.101867 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance
I20260928 08:32:38.102335 15899 external_mini_cluster.cc:1428] Setting key e5958afec9c8e58f1a8096d15455a655
Sep 28 08:32:38 dist-test-slave-qp2n krb5kdc[26731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584357, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:32:38.124691 27700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:32:38.102790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:47717 (local address 127.15.134.254:37799)
0928 08:32:38.103356 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0928 08:32:38.103361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:32:38.105102 (+ 1741us) server_negotiation.cc:407] Connection header received
0928 08:32:38.107904 (+ 2802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:32:38.107909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:32:38.108037 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:32:38.108158 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:32:38.110166 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.111089 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.113056 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.113577 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.113735 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:32:38.118303 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:32:38.118308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:32:38.118312 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:32:38.118347 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:32:38.121290 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:32:38.122094 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:32:38.122105 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:32:38.122108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:32:38.122250 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:32:38.122870 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:32:38.122875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:32:38.122878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:32:38.123167 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:32:38.123332 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 08:32:38.124043 (+ 711us) server_negotiation.cc:299] Negotiation successful
0928 08:32:38.124280 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":329,"threads_started":1}
I20260928 08:32:38.125808 27701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:32:38.103590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:37799 (local address 127.15.134.195:47717)
0928 08:32:38.104958 (+ 1368us) negotiation.cc:107] Waiting for socket to connect
0928 08:32:38.105009 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 08:32:38.107605 (+ 2596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:32:38.108417 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:32:38.108427 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:32:38.108996 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:32:38.109973 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:32:38.109998 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.111328 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.111332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:32:38.112875 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:32:38.112889 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.113837 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.113845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:32:38.114189 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:32:38.114966 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:32:38.114991 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:32:38.118053 (+ 3062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:32:38.121432 (+ 3379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:32:38.121443 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:32:38.121470 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:32:38.121959 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:32:38.122434 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:32:38.122438 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:32:38.122441 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:32:38.122742 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:32:38.123308 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:32:38.123316 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:32:38.123661 (+ 345us) client_negotiation.cc:769] Sending connection context
0928 08:32:38.123943 (+ 282us) client_negotiation.cc:240] Negotiation successful
0928 08:32:38.124464 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1028,"thread_start_us":673,"threads_started":1}
I20260928 08:32:38.127170 27698 heartbeater.cc:347] Connected to a master server at 127.15.134.254:37799
I20260928 08:32:38.127513 27698 heartbeater.cc:464] Registering TS with master...
I20260928 08:32:38.128304 27698 heartbeater.cc:511] Master 127.15.134.254:37799 requested a full tablet report, sending...
I20260928 08:32:38.130126 27208 ts_manager.cc:194] Registered new tserver with Master: 9dccaa9db2254fe59bfbf4b54a6519fe (127.15.134.195:38195)
I20260928 08:32:38.131194 27208 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.195@KRBTEST.COM'} at 127.15.134.195:47717
I20260928 08:32:38.132231 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:32:38.152349 27700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:32:38.137849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40422 (local address 127.15.134.254:37799)
0928 08:32:38.138088 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 08:32:38.138097 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:32:38.138845 (+ 748us) server_negotiation.cc:407] Connection header received
0928 08:32:38.139111 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:32:38.139115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:32:38.139188 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:32:38.139299 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:32:38.140977 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.141713 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.143200 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.143648 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.143791 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:32:38.146492 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:32:38.146497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:32:38.146501 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:32:38.146543 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:32:38.149332 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:32:38.150289 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:32:38.150297 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:32:38.150300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:32:38.150432 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:32:38.150868 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:32:38.150874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:32:38.150878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:32:38.151171 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:32:38.151434 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 08:32:38.151778 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 08:32:38.152014 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 08:32:38.157078 27207 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:32:38.165071 27207 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:32:38.197422 27713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:32:38.179293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:38195 (local address 127.0.0.1:59828)
0928 08:32:38.181095 (+ 1802us) negotiation.cc:107] Waiting for socket to connect
0928 08:32:38.181120 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 08:32:38.181254 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:32:38.184161 (+ 2907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:32:38.184173 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:32:38.184224 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:32:38.184806 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:32:38.184825 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.188642 (+ 3817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.188651 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:32:38.193550 (+ 4899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:32:38.193573 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.196484 (+ 2911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.196489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:32:38.196669 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:32:38.196714 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 08:32:38.196845 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 08:32:38.197046 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1628,"thread_start_us":1633,"threads_started":1}
I20260928 08:32:38.197422 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:32:38.179729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59828 (local address 127.15.134.195:38195)
0928 08:32:38.183018 (+ 3289us) server_negotiation.cc:206] Beginning negotiation
0928 08:32:38.183029 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:32:38.183078 (+ 49us) server_negotiation.cc:407] Connection header received
0928 08:32:38.183196 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:32:38.183208 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:32:38.183530 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:32:38.183756 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:32:38.185085 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.188421 (+ 3336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.193836 (+ 5415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.195947 (+ 2111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.196154 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:32:38.196358 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 08:32:38.196901 (+ 543us) server_negotiation.cc:299] Negotiation successful
0928 08:32:38.197061 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3042,"thread_start_us":2898,"threads_started":1}
I20260928 08:32:38.202702 27712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:32:38.178648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:41859 (local address 127.0.0.1:50016)
0928 08:32:38.180120 (+ 1472us) negotiation.cc:107] Waiting for socket to connect
0928 08:32:38.180176 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 08:32:38.180392 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:32:38.194549 (+ 14157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:32:38.194554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:32:38.194579 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:32:38.195366 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:32:38.195381 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.197834 (+ 2453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.197840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:32:38.200007 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:32:38.200019 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.201857 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.201864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:32:38.202078 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:32:38.202113 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 08:32:38.202186 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 08:32:38.202318 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1136,"mutex_wait_us":20,"thread_start_us":980,"threads_started":1}
I20260928 08:32:38.202593 27633 tablet_service.cc:1467] Processing CreateTablet for tablet b5cb6d59a3ff45618d473f51240610fc (DEFAULT_TABLE table=test-table [id=d118244c8ee145a195d6bce192ab1dbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:32:38.203928 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:32:38.178846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50016 (local address 127.15.134.193:41859)
0928 08:32:38.194010 (+ 15164us) server_negotiation.cc:206] Beginning negotiation
0928 08:32:38.194019 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:32:38.194045 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:32:38.194131 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:32:38.194137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:32:38.194338 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:32:38.194577 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:32:38.195596 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.197636 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.200163 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.201570 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.203153 (+ 1583us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:32:38.203329 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 08:32:38.203416 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 08:32:38.203580 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14982,"thread_start_us":14795,"threads_started":1}
I20260928 08:32:38.204246 27716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:32:38.182440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55920 (local address 127.15.134.194:44807)
0928 08:32:38.194949 (+ 12509us) server_negotiation.cc:206] Beginning negotiation
0928 08:32:38.194957 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:32:38.194985 (+ 28us) server_negotiation.cc:407] Connection header received
0928 08:32:38.195071 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:32:38.195077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:32:38.195347 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:32:38.196492 (+ 1145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:32:38.196748 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.198311 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.200819 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.202821 (+ 2002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.203013 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:32:38.203184 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 08:32:38.203708 (+ 524us) server_negotiation.cc:299] Negotiation successful
0928 08:32:38.203897 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12312,"thread_start_us":13055,"threads_started":1}
I20260928 08:32:38.204355 27715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:32:38.181882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:44807 (local address 127.0.0.1:55920)
0928 08:32:38.192962 (+ 11080us) negotiation.cc:107] Waiting for socket to connect
0928 08:32:38.192985 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:32:38.193132 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:32:38.195753 (+ 2621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:32:38.195757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:32:38.195793 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:32:38.196275 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:32:38.196287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.198443 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.198451 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:32:38.200575 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:32:38.200589 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.203332 (+ 2743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.203338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:32:38.203522 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:32:38.203555 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 08:32:38.203651 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 08:32:38.203785 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10900,"mutex_wait_us":3,"thread_start_us":11354,"threads_started":1}
I20260928 08:32:38.204443 27633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5cb6d59a3ff45618d473f51240610fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:32:38.207726 27499 tablet_service.cc:1467] Processing CreateTablet for tablet b5cb6d59a3ff45618d473f51240610fc (DEFAULT_TABLE table=test-table [id=d118244c8ee145a195d6bce192ab1dbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:32:38.208125 27356 tablet_service.cc:1467] Processing CreateTablet for tablet b5cb6d59a3ff45618d473f51240610fc (DEFAULT_TABLE table=test-table [id=d118244c8ee145a195d6bce192ab1dbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:32:38.209147 27499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5cb6d59a3ff45618d473f51240610fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:32:38.210079 27356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5cb6d59a3ff45618d473f51240610fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:32:38.220533 27717 tablet_bootstrap.cc:492] T b5cb6d59a3ff45618d473f51240610fc P f74a37b2b4494135a206d5604c9ede6f: Bootstrap starting.
I20260928 08:32:38.221517 27718 tablet_bootstrap.cc:492] T b5cb6d59a3ff45618d473f51240610fc P 9dccaa9db2254fe59bfbf4b54a6519fe: Bootstrap starting.
I20260928 08:32:38.224941 27717 tablet_bootstrap.cc:654] T b5cb6d59a3ff45618d473f51240610fc P f74a37b2b4494135a206d5604c9ede6f: Neither blocks nor log segments found. Creating new log.
I20260928 08:32:38.225733 27718 tablet_bootstrap.cc:654] T b5cb6d59a3ff45618d473f51240610fc P 9dccaa9db2254fe59bfbf4b54a6519fe: Neither blocks nor log segments found. Creating new log.
I20260928 08:32:38.226594 27717 log.cc:824] T b5cb6d59a3ff45618d473f51240610fc P f74a37b2b4494135a206d5604c9ede6f: Log is configured to *not* fsync() on all Append() calls
I20260928 08:32:38.226878 27718 log.cc:824] T b5cb6d59a3ff45618d473f51240610fc P 9dccaa9db2254fe59bfbf4b54a6519fe: Log is configured to *not* fsync() on all Append() calls
I20260928 08:32:38.227254 27719 tablet_bootstrap.cc:492] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35: Bootstrap starting.
I20260928 08:32:38.230609 27719 tablet_bootstrap.cc:654] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35: Neither blocks nor log segments found. Creating new log.
I20260928 08:32:38.231554 27719 log.cc:824] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35: Log is configured to *not* fsync() on all Append() calls
I20260928 08:32:38.235000 27719 tablet_bootstrap.cc:492] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35: No bootstrap required, opened a new log
I20260928 08:32:38.235252 27717 tablet_bootstrap.cc:492] T b5cb6d59a3ff45618d473f51240610fc P f74a37b2b4494135a206d5604c9ede6f: No bootstrap required, opened a new log
I20260928 08:32:38.235250 27718 tablet_bootstrap.cc:492] T b5cb6d59a3ff45618d473f51240610fc P 9dccaa9db2254fe59bfbf4b54a6519fe: No bootstrap required, opened a new log
I20260928 08:32:38.235327 27719 ts_tablet_manager.cc:1397] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 08:32:38.235526 27718 ts_tablet_manager.cc:1397] T b5cb6d59a3ff45618d473f51240610fc P 9dccaa9db2254fe59bfbf4b54a6519fe: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20260928 08:32:38.235584 27717 ts_tablet_manager.cc:1397] T b5cb6d59a3ff45618d473f51240610fc P f74a37b2b4494135a206d5604c9ede6f: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.011s
I20260928 08:32:38.244108 27719 raft_consensus.cc:348] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817100cfa3b9403398bfb15320a0db35" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 41859 } } peers { permanent_uuid: "9dccaa9db2254fe59bfbf4b54a6519fe" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38195 } } peers { permanent_uuid: "f74a37b2b4494135a206d5604c9ede6f" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44807 } }
I20260928 08:32:38.244544 27719 raft_consensus.cc:374] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:32:38.244709 27719 raft_consensus.cc:729] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 817100cfa3b9403398bfb15320a0db35, State: Initialized, Role: FOLLOWER
I20260928 08:32:38.245395 27719 consensus_queue.cc:260] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817100cfa3b9403398bfb15320a0db35" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 41859 } } peers { permanent_uuid: "9dccaa9db2254fe59bfbf4b54a6519fe" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38195 } } peers { permanent_uuid: "f74a37b2b4494135a206d5604c9ede6f" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44807 } }
I20260928 08:32:38.246908 27717 raft_consensus.cc:348] T b5cb6d59a3ff45618d473f51240610fc P f74a37b2b4494135a206d5604c9ede6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817100cfa3b9403398bfb15320a0db35" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 41859 } } peers { permanent_uuid: "9dccaa9db2254fe59bfbf4b54a6519fe" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38195 } } peers { permanent_uuid: "f74a37b2b4494135a206d5604c9ede6f" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44807 } }
I20260928 08:32:38.247354 27719 ts_tablet_manager.cc:1428] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
I20260928 08:32:38.247496 27717 raft_consensus.cc:374] T b5cb6d59a3ff45618d473f51240610fc P f74a37b2b4494135a206d5604c9ede6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:32:38.247254 27718 raft_consensus.cc:348] T b5cb6d59a3ff45618d473f51240610fc P 9dccaa9db2254fe59bfbf4b54a6519fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817100cfa3b9403398bfb15320a0db35" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 41859 } } peers { permanent_uuid: "9dccaa9db2254fe59bfbf4b54a6519fe" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38195 } } peers { permanent_uuid: "f74a37b2b4494135a206d5604c9ede6f" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44807 } }
I20260928 08:32:38.247680 27717 raft_consensus.cc:729] T b5cb6d59a3ff45618d473f51240610fc P f74a37b2b4494135a206d5604c9ede6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74a37b2b4494135a206d5604c9ede6f, State: Initialized, Role: FOLLOWER
I20260928 08:32:38.247792 27718 raft_consensus.cc:374] T b5cb6d59a3ff45618d473f51240610fc P 9dccaa9db2254fe59bfbf4b54a6519fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:32:38.247937 27718 raft_consensus.cc:729] T b5cb6d59a3ff45618d473f51240610fc P 9dccaa9db2254fe59bfbf4b54a6519fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dccaa9db2254fe59bfbf4b54a6519fe, State: Initialized, Role: FOLLOWER
I20260928 08:32:38.248525 27717 consensus_queue.cc:260] T b5cb6d59a3ff45618d473f51240610fc P f74a37b2b4494135a206d5604c9ede6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817100cfa3b9403398bfb15320a0db35" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 41859 } } peers { permanent_uuid: "9dccaa9db2254fe59bfbf4b54a6519fe" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38195 } } peers { permanent_uuid: "f74a37b2b4494135a206d5604c9ede6f" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44807 } }
I20260928 08:32:38.248890 27718 consensus_queue.cc:260] T b5cb6d59a3ff45618d473f51240610fc P 9dccaa9db2254fe59bfbf4b54a6519fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817100cfa3b9403398bfb15320a0db35" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 41859 } } peers { permanent_uuid: "9dccaa9db2254fe59bfbf4b54a6519fe" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38195 } } peers { permanent_uuid: "f74a37b2b4494135a206d5604c9ede6f" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44807 } }
I20260928 08:32:38.250146 27564 heartbeater.cc:503] Master 127.15.134.254:37799 was elected leader, sending a full tablet report...
I20260928 08:32:38.251187 27717 ts_tablet_manager.cc:1428] T b5cb6d59a3ff45618d473f51240610fc P f74a37b2b4494135a206d5604c9ede6f: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260928 08:32:38.254293 27698 heartbeater.cc:503] Master 127.15.134.254:37799 was elected leader, sending a full tablet report...
I20260928 08:32:38.256472 27718 ts_tablet_manager.cc:1428] T b5cb6d59a3ff45618d473f51240610fc P 9dccaa9db2254fe59bfbf4b54a6519fe: Time spent starting tablet: real 0.021s user 0.019s sys 0.001s
I20260928 08:32:38.262916 27723 raft_consensus.cc:482] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:32:38.263239 27723 raft_consensus.cc:504] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817100cfa3b9403398bfb15320a0db35" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 41859 } } peers { permanent_uuid: "9dccaa9db2254fe59bfbf4b54a6519fe" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38195 } } peers { permanent_uuid: "f74a37b2b4494135a206d5604c9ede6f" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44807 } }
I20260928 08:32:38.265246 27723 leader_election.cc:288] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dccaa9db2254fe59bfbf4b54a6519fe (127.15.134.195:38195), f74a37b2b4494135a206d5604c9ede6f (127.15.134.194:44807)
I20260928 08:32:38.275450 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:32:38.265814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:48645 (local address 127.15.134.195:38195)
0928 08:32:38.266045 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 08:32:38.266050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:32:38.266844 (+ 794us) server_negotiation.cc:407] Connection header received
0928 08:32:38.267134 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:32:38.267140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:32:38.267212 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:32:38.267329 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:32:38.268237 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.269737 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.272694 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.274148 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.274371 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:32:38.274464 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 08:32:38.274870 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 08:32:38.275044 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 08:32:38.275588 27726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:32:38.265625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:38195 (local address 127.15.134.193:48645)
0928 08:32:38.266691 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0928 08:32:38.266712 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:32:38.266898 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:32:38.267384 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:32:38.267390 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:32:38.267449 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:32:38.268073 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:32:38.268086 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.269859 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.269862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:32:38.272473 (+ 2611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:32:38.272491 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.274385 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.274418 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:32:38.274627 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:32:38.274666 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 08:32:38.274770 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 08:32:38.274944 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"mutex_wait_us":95,"spinlock_wait_cycles":30080,"thread_start_us":648,"threads_started":1}
I20260928 08:32:38.275808 27727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:32:38.265811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:44807 (local address 127.15.134.193:41155)
0928 08:32:38.266762 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0928 08:32:38.266777 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:32:38.266925 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:32:38.267384 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:32:38.267390 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:32:38.267423 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:32:38.267895 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:32:38.267911 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.269849 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.269856 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:32:38.272473 (+ 2617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:32:38.272491 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.274531 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.274537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:32:38.274686 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:32:38.274721 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 08:32:38.274798 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 08:32:38.274944 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":64,"spinlock_wait_cycles":23936,"thread_start_us":519,"threads_started":1}
I20260928 08:32:38.276021 27716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:32:38.265795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:41155 (local address 127.15.134.194:44807)
0928 08:32:38.266000 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 08:32:38.266007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:32:38.266868 (+ 861us) server_negotiation.cc:407] Connection header received
0928 08:32:38.267125 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:32:38.267131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:32:38.267212 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:32:38.267317 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:32:38.268235 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.269670 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.272694 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.274330 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.274484 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:32:38.274556 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 08:32:38.274870 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 08:32:38.275021 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 08:32:38.276822 27653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5cb6d59a3ff45618d473f51240610fc" candidate_uuid: "817100cfa3b9403398bfb15320a0db35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dccaa9db2254fe59bfbf4b54a6519fe" is_pre_election: true
I20260928 08:32:38.277716 27653 raft_consensus.cc:2454] T b5cb6d59a3ff45618d473f51240610fc P 9dccaa9db2254fe59bfbf4b54a6519fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 817100cfa3b9403398bfb15320a0db35 in term 0.
I20260928 08:32:38.277478 27519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5cb6d59a3ff45618d473f51240610fc" candidate_uuid: "817100cfa3b9403398bfb15320a0db35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74a37b2b4494135a206d5604c9ede6f" is_pre_election: true
I20260928 08:32:38.278366 27519 raft_consensus.cc:2454] T b5cb6d59a3ff45618d473f51240610fc P f74a37b2b4494135a206d5604c9ede6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 817100cfa3b9403398bfb15320a0db35 in term 0.
I20260928 08:32:38.279171 27312 leader_election.cc:302] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 817100cfa3b9403398bfb15320a0db35, f74a37b2b4494135a206d5604c9ede6f; no voters:
I20260928 08:32:38.279670 27723 raft_consensus.cc:2781] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:32:38.279843 27723 raft_consensus.cc:482] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:32:38.279979 27723 raft_consensus.cc:3037] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:32:38.283176 27723 raft_consensus.cc:504] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817100cfa3b9403398bfb15320a0db35" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 41859 } } peers { permanent_uuid: "9dccaa9db2254fe59bfbf4b54a6519fe" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38195 } } peers { permanent_uuid: "f74a37b2b4494135a206d5604c9ede6f" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44807 } }
I20260928 08:32:38.284246 27723 leader_election.cc:288] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35 [CANDIDATE]: Term 1 election: Requested vote from peers 9dccaa9db2254fe59bfbf4b54a6519fe (127.15.134.195:38195), f74a37b2b4494135a206d5604c9ede6f (127.15.134.194:44807)
I20260928 08:32:38.284677 27653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5cb6d59a3ff45618d473f51240610fc" candidate_uuid: "817100cfa3b9403398bfb15320a0db35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dccaa9db2254fe59bfbf4b54a6519fe"
I20260928 08:32:38.284798 27519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5cb6d59a3ff45618d473f51240610fc" candidate_uuid: "817100cfa3b9403398bfb15320a0db35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f74a37b2b4494135a206d5604c9ede6f"
I20260928 08:32:38.284904 27653 raft_consensus.cc:3037] T b5cb6d59a3ff45618d473f51240610fc P 9dccaa9db2254fe59bfbf4b54a6519fe [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:32:38.285120 27519 raft_consensus.cc:3037] T b5cb6d59a3ff45618d473f51240610fc P f74a37b2b4494135a206d5604c9ede6f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:32:38.287350 27653 raft_consensus.cc:2454] T b5cb6d59a3ff45618d473f51240610fc P 9dccaa9db2254fe59bfbf4b54a6519fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 817100cfa3b9403398bfb15320a0db35 in term 1.
I20260928 08:32:38.287583 27519 raft_consensus.cc:2454] T b5cb6d59a3ff45618d473f51240610fc P f74a37b2b4494135a206d5604c9ede6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 817100cfa3b9403398bfb15320a0db35 in term 1.
I20260928 08:32:38.287959 27310 leader_election.cc:302] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 817100cfa3b9403398bfb15320a0db35, 9dccaa9db2254fe59bfbf4b54a6519fe; no voters:
I20260928 08:32:38.288501 27723 raft_consensus.cc:2781] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:32:38.289309 27723 raft_consensus.cc:686] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35 [term 1 LEADER]: Becoming Leader. State: Replica: 817100cfa3b9403398bfb15320a0db35, State: Running, Role: LEADER
I20260928 08:32:38.289899 27723 consensus_queue.cc:237] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817100cfa3b9403398bfb15320a0db35" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 41859 } } peers { permanent_uuid: "9dccaa9db2254fe59bfbf4b54a6519fe" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38195 } } peers { permanent_uuid: "f74a37b2b4494135a206d5604c9ede6f" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44807 } }
I20260928 08:32:38.295410 27208 catalog_manager.cc:5495] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 817100cfa3b9403398bfb15320a0db35 (127.15.134.193). New cstate: current_term: 1 leader_uuid: "817100cfa3b9403398bfb15320a0db35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "817100cfa3b9403398bfb15320a0db35" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 41859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9dccaa9db2254fe59bfbf4b54a6519fe" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74a37b2b4494135a206d5604c9ede6f" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44807 } health_report { overall_health: UNKNOWN } } }
I20260928 08:32:38.316347 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:32:38.310567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50020 (local address 127.15.134.193:41859)
0928 08:32:38.310761 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 08:32:38.310766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:32:38.311080 (+ 314us) server_negotiation.cc:407] Connection header received
0928 08:32:38.311263 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:32:38.311267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:32:38.311345 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:32:38.311467 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:32:38.311950 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.312926 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.314303 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.314701 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.314793 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:32:38.315276 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:32:38.315782 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:32:38.315901 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 08:32:38.316062 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 08:32:38.316157 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 08:32:38.325765 27519 raft_consensus.cc:1261] T b5cb6d59a3ff45618d473f51240610fc P f74a37b2b4494135a206d5604c9ede6f [term 1 FOLLOWER]: Refusing update from remote peer 817100cfa3b9403398bfb15320a0db35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:32:38.325901 27653 raft_consensus.cc:1261] T b5cb6d59a3ff45618d473f51240610fc P 9dccaa9db2254fe59bfbf4b54a6519fe [term 1 FOLLOWER]: Refusing update from remote peer 817100cfa3b9403398bfb15320a0db35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:32:38.326980 27723 consensus_queue.cc:1035] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f74a37b2b4494135a206d5604c9ede6f" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:32:38.327392 27728 consensus_queue.cc:1035] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dccaa9db2254fe59bfbf4b54a6519fe" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:32:38.339013 27731 mvcc.cc:205] Tried to move back new op lower bound from 7334233531691245568 to 7334233531560062976. Current Snapshot: MvccSnapshot[applied={T|T < 7334233531691245568}]
I20260928 08:32:38.339426 27732 mvcc.cc:205] Tried to move back new op lower bound from 7334233531691245568 to 7334233531560062976. Current Snapshot: MvccSnapshot[applied={T|T < 7334233531691245568}]
I20260928 08:32:38.356099 27208 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:32:38.356392 27208 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:32:38.360235 27208 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2867122358a45e9ac6c47c86659919c: Sending DeleteTablet for 3 replicas of tablet b5cb6d59a3ff45618d473f51240610fc
I20260928 08:32:38.361644 27356 tablet_service.cc:1514] Processing DeleteTablet for tablet b5cb6d59a3ff45618d473f51240610fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:32:38 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:50016
I20260928 08:32:38.361683 27633 tablet_service.cc:1514] Processing DeleteTablet for tablet b5cb6d59a3ff45618d473f51240610fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:32:38 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:59828
I20260928 08:32:38.362601 27741 tablet_replica.cc:330] T b5cb6d59a3ff45618d473f51240610fc P 9dccaa9db2254fe59bfbf4b54a6519fe: stopping tablet replica
I20260928 08:32:38.362715 27499 tablet_service.cc:1514] Processing DeleteTablet for tablet b5cb6d59a3ff45618d473f51240610fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:32:38 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:55920
I20260928 08:32:38.363106 27741 raft_consensus.cc:2229] T b5cb6d59a3ff45618d473f51240610fc P 9dccaa9db2254fe59bfbf4b54a6519fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:32:38.363694 27741 raft_consensus.cc:2258] T b5cb6d59a3ff45618d473f51240610fc P 9dccaa9db2254fe59bfbf4b54a6519fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:32:38.365734 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 27288
I20260928 08:32:38.375221 27741 ts_tablet_manager.cc:1905] T b5cb6d59a3ff45618d473f51240610fc P 9dccaa9db2254fe59bfbf4b54a6519fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:32:38.381994 27743 tablet_replica.cc:330] T b5cb6d59a3ff45618d473f51240610fc P f74a37b2b4494135a206d5604c9ede6f: stopping tablet replica
I20260928 08:32:38.382859 27743 raft_consensus.cc:2229] T b5cb6d59a3ff45618d473f51240610fc P f74a37b2b4494135a206d5604c9ede6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:32:38.383863 27743 raft_consensus.cc:2258] T b5cb6d59a3ff45618d473f51240610fc P f74a37b2b4494135a206d5604c9ede6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:32:38.391894 27741 ts_tablet_manager.cc:1918] T b5cb6d59a3ff45618d473f51240610fc P 9dccaa9db2254fe59bfbf4b54a6519fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:32:38.393332 27742 tablet_replica.cc:330] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35: stopping tablet replica
I20260928 08:32:38.393951 27742 raft_consensus.cc:2229] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:32:38.395154 27742 raft_consensus.cc:2258] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:32:38.395248 27743 ts_tablet_manager.cc:1905] T b5cb6d59a3ff45618d473f51240610fc P f74a37b2b4494135a206d5604c9ede6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:32:38.397398 27742 ts_tablet_manager.cc:1905] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:32:38.403126 27741 log.cc:1196] T b5cb6d59a3ff45618d473f51240610fc P 9dccaa9db2254fe59bfbf4b54a6519fe: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/b5cb6d59a3ff45618d473f51240610fc
I20260928 08:32:38.403901 27741 ts_tablet_manager.cc:1939] T b5cb6d59a3ff45618d473f51240610fc P 9dccaa9db2254fe59bfbf4b54a6519fe: Deleting consensus metadata
I20260928 08:32:38.406270 27743 ts_tablet_manager.cc:1918] T b5cb6d59a3ff45618d473f51240610fc P f74a37b2b4494135a206d5604c9ede6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:32:38.407058 27742 ts_tablet_manager.cc:1918] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:32:38.407306 27742 log.cc:1196] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-0/wal/wals/b5cb6d59a3ff45618d473f51240610fc
I20260928 08:32:38.407913 27742 ts_tablet_manager.cc:1939] T b5cb6d59a3ff45618d473f51240610fc P 817100cfa3b9403398bfb15320a0db35: Deleting consensus metadata
Sep 28 08:32:38 dist-test-slave-qp2n krb5kdc[26731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.193@KRBTEST.COM
I20260928 08:32:38.408409 27193 catalog_manager.cc:4841] TS 9dccaa9db2254fe59bfbf4b54a6519fe (127.15.134.195:38195): tablet b5cb6d59a3ff45618d473f51240610fc (table test-table [id=d118244c8ee145a195d6bce192ab1dbc]) successfully deleted
I20260928 08:32:38.409087 27743 log.cc:1196] T b5cb6d59a3ff45618d473f51240610fc P f74a37b2b4494135a206d5604c9ede6f: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/b5cb6d59a3ff45618d473f51240610fc
I20260928 08:32:38.409693 27743 ts_tablet_manager.cc:1939] T b5cb6d59a3ff45618d473f51240610fc P f74a37b2b4494135a206d5604c9ede6f: Deleting consensus metadata
I20260928 08:32:38.409848 27194 catalog_manager.cc:4841] TS 817100cfa3b9403398bfb15320a0db35 (127.15.134.193:41859): tablet b5cb6d59a3ff45618d473f51240610fc (table test-table [id=d118244c8ee145a195d6bce192ab1dbc]) successfully deleted
I20260928 08:32:38.411655 27195 catalog_manager.cc:4841] TS f74a37b2b4494135a206d5604c9ede6f (127.15.134.194:44807): tablet b5cb6d59a3ff45618d473f51240610fc (table test-table [id=d118244c8ee145a195d6bce192ab1dbc]) successfully deleted
I20260928 08:32:38.416917 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:32:38.367380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50028 (local address 127.15.134.193:41859)
0928 08:32:38.367607 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 08:32:38.367615 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:32:38.396937 (+ 29322us) server_negotiation.cc:407] Connection header received
0928 08:32:38.397192 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:32:38.397198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:32:38.397290 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:32:38.397390 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:32:38.399507 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.400811 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.402827 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.403585 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.403709 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:32:38.409560 (+ 5851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:32:38.409581 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:32:38.409600 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:32:38.409662 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:32:38.413451 (+ 3789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:32:38.414269 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:32:38.414276 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:32:38.414279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:32:38.414428 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:32:38.415046 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:32:38.415052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:32:38.415055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:32:38.415434 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:32:38.415774 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0928 08:32:38.416094 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 08:32:38.416346 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:32:38.421123 27416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:32:38.528816 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 27288
I20260928 08:32:38.549034 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 27427
Sep 28 08:32:38 dist-test-slave-qp2n krb5kdc[26731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.194@KRBTEST.COM
I20260928 08:32:38.566334 27716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:32:38.549872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55924 (local address 127.15.134.194:44807)
0928 08:32:38.550082 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 08:32:38.550089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:32:38.550206 (+ 117us) server_negotiation.cc:407] Connection header received
0928 08:32:38.550438 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:32:38.550451 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:32:38.550516 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:32:38.550644 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:32:38.551976 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.552809 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.554139 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.554524 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.554662 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:32:38.559628 (+ 4966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:32:38.559653 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:32:38.559676 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:32:38.559713 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:32:38.563156 (+ 3443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:32:38.563933 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:32:38.563950 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:32:38.563953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:32:38.564082 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:32:38.564517 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:32:38.564528 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:32:38.564531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:32:38.565083 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:32:38.565375 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0928 08:32:38.565759 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 08:32:38.565975 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 08:32:38.567029 27559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:32:38.658741 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 27427
I20260928 08:32:38.676394 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 27570
Sep 28 08:32:38 dist-test-slave-qp2n krb5kdc[26731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.195@KRBTEST.COM
I20260928 08:32:38.693190 27714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:32:38.677418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59842 (local address 127.15.134.195:38195)
0928 08:32:38.677621 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 08:32:38.677628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:32:38.677739 (+ 111us) server_negotiation.cc:407] Connection header received
0928 08:32:38.677958 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:32:38.677964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:32:38.678049 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:32:38.678175 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:32:38.679628 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.680446 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.681947 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:32:38.682343 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:32:38.682483 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:32:38.686939 (+ 4456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:32:38.686955 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:32:38.686973 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:32:38.687007 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:32:38.690221 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:32:38.690989 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:32:38.691003 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:32:38.691005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:32:38.691137 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:32:38.691553 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:32:38.691559 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:32:38.691562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:32:38.691939 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:32:38.692250 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0928 08:32:38.692573 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 08:32:38.692850 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 08:32:38.693980 27693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:32:38.786556 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 27570
I20260928 08:32:38.805102 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 27176
I20260928 08:32:38.806361 27238 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:32:38.922870 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 27176
I20260928 08:32:38.947149 15899 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 08:32:39.283296 15899 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 08:32:39.283620 15899 mini_ranger.cc:66] Stopping Ranger...
I20260928 08:32:39.629611 15899 mini_ranger.cc:68] Stopped Ranger
2026-09-28 08:32:39.629 UTC [26761] LOG: received smart shutdown request
2026-09-28 08:32:39.633 UTC [26761] LOG: background worker "logical replication launcher" (PID 26770) exited with exit code 1
2026-09-28 08:32:39.634 UTC [26765] LOG: shutting down
2026-09-28 08:32:39.834 UTC [26761] LOG: database system is shut down
2026-09-28T08:32:39Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (54782 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:32:40 dist-test-slave-qp2n krb5kdc[27757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:32:40 dist-test-slave-qp2n krb5kdc[27757](info): set up 2 sockets
Sep 28 08:32:40 dist-test-slave-qp2n krb5kdc[27757](info): commencing operation
krb5kdc: starting...
W20260928 08:32:41.090664 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:32:41 dist-test-slave-qp2n krb5kdc[27757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:32:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:32:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.15.134.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.15.134.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.15.134.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/rangeradmin_127.15.134.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.15.134.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/rangeradmin_127.15.134.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.15.134.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.15.134.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.15.134.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/rangerlookup_127.15.134.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.15.134.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/rangerlookup_127.15.134.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.134.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.15.134.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.134.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/HTTP_127.15.134.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.134.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/HTTP_127.15.134.212@KRBTEST.COM.keytab.
I20260928 08:32:41.432015 15899 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task0uZKZe/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/postgres -l logfile start
2026-09-28 08:32:44.096 UTC [27787] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 08:32:44.096 UTC [27787] LOG: listening on IPv4 address "127.15.134.212", port 54429
2026-09-28 08:32:44.100 UTC [27787] LOG: listening on Unix socket "/tmp/.s.PGSQL.54429"
2026-09-28 08:32:44.122 UTC [27790] LOG: database system was shut down at 2026-09-28 08:32:42 UTC
2026-09-28 08:32:44.126 UTC [27787] LOG: database system is ready to accept connections
I20260928 08:32:45.126086 15899 mini_postgres.cc:96] Postgres bound to 54429
2026-09-28 08:32:45.143 UTC [27798] FATAL: database "slave" does not exist
127.15.134.212:54429 - accepting connections
I20260928 08:32:45.144785 15899 mini_ranger.cc:161] Starting Ranger...
I20260928 08:32:45.172022 15899 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 08:32:45.721464 15899 mini_ranger.cc:87] Created ranger Postgres database
I20260928 08:32:45.721554 15899 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ranger-admin
2026-09-28 08:32:46,003 [I] DB FLAVOR :POSTGRES
2026-09-28 08:32:46,004 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 08:32:46,004 [I] Checking connection..
2026-09-28 08:32:46,004 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:54429/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 08:32:46,263 [I] Checking connection passed.
2026-09-28 08:32:46,264 [I] --------- Verifying version history table ---------
2026-09-28 08:32:46,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:54429/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 08:32:46,525 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 08:32:46,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:54429/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 08:32:46,777 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 08:32:46,777 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 08:32:46,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:54429/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 08:32:47.019 UTC [27862] WARNING: there is no transaction in progress
2026-09-28 08:32:47,026 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 08:32:47,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:54429/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 08:32:47,279 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 08:32:47,280 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 08:32:47,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:54429/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 08:32:47,532 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:54429/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 08:32:47,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:54429/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-qp2n.c.gcp-upstream.internal','N') ;"
2026-09-28 08:32:48,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:54429/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 08:32:48,266 [I] Table x_portal_user does not exist in database ranger
2026-09-28 08:32:48,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:54429/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 08:32:48,505 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 08:32:48,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:54429/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 08:32:48,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:54429/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 08:32:48,977 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 08:32:48,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:54429/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 08:32:49.986 UTC [27970] WARNING: there is no transaction in progress
2026-09-28 08:32:50.004 UTC [27970] WARNING: there is no transaction in progress
2026-09-28 08:32:50.024 UTC [27970] WARNING: there is no transaction in progress
2026-09-28 08:32:50.042 UTC [27970] WARNING: there is no transaction in progress
2026-09-28 08:32:50.060 UTC [27970] WARNING: there is no transaction in progress
2026-09-28 08:32:50.080 UTC [27970] WARNING: there is no transaction in progress
2026-09-28 08:32:50.138 UTC [27970] WARNING: there is no transaction in progress
2026-09-28 08:32:50.145 UTC [27970] WARNING: there is no transaction in progress
2026-09-28 08:32:50.161 UTC [27970] WARNING: there is no transaction in progress
2026-09-28 08:32:50.173 UTC [27970] WARNING: there is no transaction in progress
2026-09-28 08:32:50.185 UTC [27970] WARNING: there is no transaction in progress
2026-09-28 08:32:50.201 UTC [27970] WARNING: there is no transaction in progress
2026-09-28 08:32:50.212 UTC [27970] WARNING: there is no transaction in progress
2026-09-28 08:32:50.219 UTC [27970] WARNING: there is no transaction in progress
2026-09-28 08:32:50.226 UTC [27970] WARNING: there is no transaction in progress
2026-09-28 08:32:50.233 UTC [27970] WARNING: there is no transaction in progress
2026-09-28 08:32:50,971 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 08:32:50,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:54429/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 08:32:51,220 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 08:32:51,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:54429/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 08:32:51,485 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 08:32:51,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:54429/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 08:32:51,754 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 08:32:51,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:54429/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 08:32:51,992 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:54429/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 08:32:52,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:54429/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-qp2n.c.gcp-upstream.internal';"
2026-09-28 08:32:52,521 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 08:32:52,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:54429/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 08:32:52,773 [I] DB_PATCHES have already been applied
I20260928 08:32:52.779274 15899 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ranger-admin:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task0uZKZe/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 08:32:52.779471 15899 mini_ranger.cc:193] Using host: 127.15.134.212
I20260928 08:32:52.794970 15899 mini_ranger.cc:227] Ranger admin URL: http://127.15.134.212:46317
Sep 28, 2026 8:32:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 8:32:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 8:32:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 8:32:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 8:32:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 8:32:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 8:32:53 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 08:32:53.989652 15899 mini_ranger.cc:160] Time spent starting Ranger: real 8.845s user 0.000s sys 0.042s
Sep 28, 2026 8:33:14 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 8:33:14 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 8:33:14 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 8:33:14 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 8:33:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 8:33:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 8:33:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 8:33:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 8:33:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 8:33:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 8:33:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 8:33:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 8:33:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 8:33:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 8:33:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 8:33:14 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 8:33:14 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 8:33:15 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 08:33:16.445891 15899 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.15.134.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.15.134.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.15.134.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/rangerkms_127.15.134.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.15.134.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/rangerkms_127.15.134.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.15.134.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.15.134.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.15.134.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/HTTP_127.15.134.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.15.134.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/HTTP_127.15.134.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 08:33:16 dist-test-slave-qp2n krb5kdc[27757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584396, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 08:33:16.612195 15899 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 08:33:16.633569 15899 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 08:33:17.550722 15899 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 08:33:17.550812 15899 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ranger-kms
I20260928 08:33:17.550843 15899 mini_ranger_kms.cc:227] Using postgres at 127.15.134.212:54429
2026-09-28 08:33:17,908 [I] DB FLAVOR :POSTGRES
2026-09-28 08:33:17,909 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 08:33:17,909 [I] Checking connection
2026-09-28 08:33:17,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:54429/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 08:33:18,155 [I] connection success
2026-09-28 08:33:18,155 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 08:33:18,155 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 08:33:18,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:54429/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 08:33:18,392 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 08:33:18,392 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 08:33:18,392 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 08:33:18,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.15.134.212:54429/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 08:33:18,706 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 08:33:19.233187 15899 mini_ranger_kms.cc:310] Created kms service
I20260928 08:33:19.436578 15899 mini_ranger_kms.cc:326] Created kudu user
I20260928 08:33:19.481135 15899 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 08:33:19 dist-test-slave-qp2n krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584396, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.15.134.212@KRBTEST.COM
I20260928 08:33:19.939297 15899 mini_ranger_kms.cc:384] Added ranger policy
I20260928 08:33:19.939620 15899 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ranger-kms:/tmp/dist-test-task0uZKZe/build/asan/bin/postgresql.jar:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task0uZKZe/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 08:33:19.939684 15899 mini_ranger_kms.cc:240] Using host: 127.15.134.212
I20260928 08:33:19.961974 15899 mini_ranger_kms.cc:276] Ranger KMS PID: 28151
I20260928 08:33:19.962136 15899 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.15.134.212:35389
Sep 28, 2026 8:33:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 8:33:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 8:33:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 8:33:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 8:33:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 8:33:21 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 08:33:22.284538 15899 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.672s user 0.093s sys 0.284s
I20260928 08:33:22.284793 15899 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 08:33:22.284956 15899 mini_ranger_kms.cc:401] 127.15.134.212:35389/kms/v1/keys
Sep 28 08:33:22 dist-test-slave-qp2n krb5kdc[27757](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790584402, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.134.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 08:33:22 dist-test-slave-qp2n krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584402, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.134.212@KRBTEST.COM for HTTP/127.15.134.212@KRBTEST.COM
Sep 28 08:33:23 dist-test-slave-qp2n krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584402, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.15.134.212@KRBTEST.COM for HTTP/127.15.134.212@KRBTEST.COM
Sep 28 08:33:25 dist-test-slave-qp2n krb5kdc[27757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:33:25.620379 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:34443
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:34443
--ranger_config_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.134.212:35389/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:33:25.971961 28201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:33:25.972321 28201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:33:25.972483 28201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:33:25.980122 28201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:33:25.980284 28201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:33:25.980386 28201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:33:25.980474 28201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:33:25.980548 28201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:33:25.991045 28201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/wal
--ranger_kms_url=127.15.134.212:35389/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:34443
--ranger_config_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:34443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:33:25.992465 28201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:33:25.993880 28201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:33:26.008908 28209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:26.008941 28207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:26.009245 28206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:26.010319 28201 server_base.cc:1034] running on GCE node
I20260928 08:33:26.011117 28201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:33:26.012719 28201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:33:26.013938 28201 hybrid_clock.cc:648] HybridClock initialized: now 1790584406013889 us; error 44 us; skew 500 ppm
I20260928 08:33:26.014309 28201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:33:26 dist-test-slave-qp2n krb5kdc[27757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584406, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:33:26.017467 28201 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:33:26.019074 28201 webserver.cc:460] Webserver started at http://127.15.134.254:43051/ using document root <none> and password file <none>
I20260928 08:33:26.019723 28201 fs_manager.cc:359] Metadata directory not provided
I20260928 08:33:26.019815 28201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:33:26.020166 28201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:33:26 dist-test-slave-qp2n krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584406, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for HTTP/127.15.134.212@KRBTEST.COM
I20260928 08:33:26.078001 28201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "9e343dcafe71485180a75c406a699a88"
format_stamp: "Formatted at 2026-09-28 08:33:26 on dist-test-slave-qp2n"
server_key: "379c41b05001b6e6e894847083fe0c72"
server_key_iv: "ec7bf14fc4f51a6b3aaae06443a3ed2b"
server_key_version: "kuduclusterkey@0"
I20260928 08:33:26.078943 28201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "9e343dcafe71485180a75c406a699a88"
format_stamp: "Formatted at 2026-09-28 08:33:26 on dist-test-slave-qp2n"
server_key: "379c41b05001b6e6e894847083fe0c72"
server_key_iv: "ec7bf14fc4f51a6b3aaae06443a3ed2b"
server_key_version: "kuduclusterkey@0"
I20260928 08:33:26.084583 28201 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 08:33:26.102427 28216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:26.104038 28201 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 08:33:26.104228 28201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "9e343dcafe71485180a75c406a699a88"
format_stamp: "Formatted at 2026-09-28 08:33:26 on dist-test-slave-qp2n"
server_key: "379c41b05001b6e6e894847083fe0c72"
server_key_iv: "ec7bf14fc4f51a6b3aaae06443a3ed2b"
server_key_version: "kuduclusterkey@0"
I20260928 08:33:26.104473 28201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:33:26.130290 28201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:33:26.134521 28201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:33:26.134886 28201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:33:26.159482 28201 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:34443
I20260928 08:33:26.159500 28268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:34443 every 8 connection(s)
I20260928 08:33:26.160996 28201 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:33:26.165215 28269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:33:26.169322 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 28201
I20260928 08:33:26.169564 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:33:26.170269 15899 external_mini_cluster.cc:1428] Setting key 1db66b9a7a2b9cccc2beae5aa9d42658
I20260928 08:33:26.176997 28269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88: Bootstrap starting.
I20260928 08:33:26.180851 28269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88: Neither blocks nor log segments found. Creating new log.
I20260928 08:33:26.182102 28269 log.cc:824] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88: Log is configured to *not* fsync() on all Append() calls
I20260928 08:33:26.185930 28269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88: No bootstrap required, opened a new log
Sep 28 08:33:26 dist-test-slave-qp2n krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:33:26.194109 28269 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e343dcafe71485180a75c406a699a88" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 34443 } }
I20260928 08:33:26.194499 28269 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:33:26.194650 28269 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e343dcafe71485180a75c406a699a88, State: Initialized, Role: FOLLOWER
I20260928 08:33:26.195145 28272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:26.172442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58040 (local address 127.15.134.254:34443)
0928 08:33:26.173338 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:26.173350 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:26.173395 (+ 45us) server_negotiation.cc:407] Connection header received
0928 08:33:26.175206 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:26.175231 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:26.175656 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:26.175997 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:26.177982 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:26.179363 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:26.181379 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:26.182110 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:26.182290 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:26.187373 (+ 5083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:26.187381 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:26.187402 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:26.187444 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:26.190734 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:26.191472 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:26.191479 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:26.191485 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:26.191651 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:26.192071 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:26.192081 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:26.192083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:26.192768 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:26.193087 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:26.193538 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 08:33:26.193961 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":446,"threads_started":1}
I20260928 08:33:26.195364 28269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e343dcafe71485180a75c406a699a88" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 34443 } }
I20260928 08:33:26.195675 28269 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:33:26.195845 28269 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:33:26.196048 28269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:33:26.199198 28269 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e343dcafe71485180a75c406a699a88" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 34443 } }
I20260928 08:33:26.199790 28269 leader_election.cc:302] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e343dcafe71485180a75c406a699a88; no voters:
I20260928 08:33:26.200358 28269 leader_election.cc:288] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:33:26.200505 28274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:33:26.201349 28274 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [term 1 LEADER]: Becoming Leader. State: Replica: 9e343dcafe71485180a75c406a699a88, State: Running, Role: LEADER
I20260928 08:33:26.201874 28274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e343dcafe71485180a75c406a699a88" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 34443 } }
I20260928 08:33:26.202473 28269 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:33:26.204478 28276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e343dcafe71485180a75c406a699a88. Latest consensus state: current_term: 1 leader_uuid: "9e343dcafe71485180a75c406a699a88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e343dcafe71485180a75c406a699a88" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 34443 } } }
I20260928 08:33:26.204881 28276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [sys.catalog]: This master's current role is: LEADER
I20260928 08:33:26.204833 28275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e343dcafe71485180a75c406a699a88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e343dcafe71485180a75c406a699a88" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 34443 } } }
I20260928 08:33:26.205132 28275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [sys.catalog]: This master's current role is: LEADER
I20260928 08:33:26.208853 28280 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:33:26.214869 28280 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:33:26.215703 28269 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 08:33:26.235756 28280 catalog_manager.cc:1347] Generated new cluster ID: 9447a648bf9c48bda9211f285da479c1
I20260928 08:33:26.235924 28280 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:33:26.250145 28280 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:33:26.252285 28280 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:33:26.266950 28280 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88: Generated new TSK 0
I20260928 08:33:26.268332 28280 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.134.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
I20260928 08:33:26.989259 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.134.212:35389/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:0
--local_ip_for_outbound_sockets=127.15.134.193
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:34443
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:33:27 dist-test-slave-qp2n krb5kdc[27757](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790584407, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 08:33:27.455590 28314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:33:27.456012 28314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:33:27.456243 28314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:33:27.468761 28314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:33:27.468955 28314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:33:27.469245 28314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:33:27.486052 28314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.15.134.212:35389/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:34443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:33:27.487993 28314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:33:27.489902 28314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:33:27.512223 28324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:27.513501 28325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:27.521354 28327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:27.521900 28314 server_base.cc:1034] running on GCE node
I20260928 08:33:27.522763 28314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:33:27.524057 28314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:33:27.525307 28314 hybrid_clock.cc:648] HybridClock initialized: now 1790584407525231 us; error 67 us; skew 500 ppm
I20260928 08:33:27.525691 28314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:33:27 dist-test-slave-qp2n krb5kdc[27757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584407, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:33:27.529706 28314 init.cc:377] Logged in from keytab as kudu/127.15.134.193@KRBTEST.COM (short username kudu)
I20260928 08:33:27.537470 28314 webserver.cc:460] Webserver started at http://127.15.134.193:41005/ using document root <none> and password file <none>
I20260928 08:33:27.538305 28314 fs_manager.cc:359] Metadata directory not provided
I20260928 08:33:27.538466 28314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:33:27.538805 28314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:33:27 dist-test-slave-qp2n krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584407, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for HTTP/127.15.134.212@KRBTEST.COM
I20260928 08:33:27.562911 28314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/data/instance:
uuid: "eaaaa27d00d3443faf6808a5cfe6d7ee"
format_stamp: "Formatted at 2026-09-28 08:33:27 on dist-test-slave-qp2n"
server_key: "9eaf1eb28c472a4870dadee52902891f"
server_key_iv: "aa0a2482e510ba165df282371c889465"
server_key_version: "kuduclusterkey@0"
I20260928 08:33:27.564152 28314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance:
uuid: "eaaaa27d00d3443faf6808a5cfe6d7ee"
format_stamp: "Formatted at 2026-09-28 08:33:27 on dist-test-slave-qp2n"
server_key: "9eaf1eb28c472a4870dadee52902891f"
server_key_iv: "aa0a2482e510ba165df282371c889465"
server_key_version: "kuduclusterkey@0"
I20260928 08:33:27.585801 28314 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.004s sys 0.012s
I20260928 08:33:27.617357 28334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:27.619597 28314 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.008s sys 0.004s
I20260928 08:33:27.619907 28314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "eaaaa27d00d3443faf6808a5cfe6d7ee"
format_stamp: "Formatted at 2026-09-28 08:33:27 on dist-test-slave-qp2n"
server_key: "9eaf1eb28c472a4870dadee52902891f"
server_key_iv: "aa0a2482e510ba165df282371c889465"
server_key_version: "kuduclusterkey@0"
I20260928 08:33:27.620273 28314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:33:27.693327 28314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:33:27.699021 28314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:33:27.699530 28314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:33:27.701011 28314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:33:27.702932 28314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:33:27.703079 28314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 08:33:27.703272 28314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:33:27.703397 28314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:27.795033 28314 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:37011
I20260928 08:33:27.795056 28446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:37011 every 8 connection(s)
I20260928 08:33:27.797017 28314 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
Sep 28 08:33:27 dist-test-slave-qp2n krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584407, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for HTTP/127.15.134.212@KRBTEST.COM
I20260928 08:33:27.802209 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 28314
I20260928 08:33:27.802412 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance
I20260928 08:33:27.803212 15899 external_mini_cluster.cc:1428] Setting key b4853498a66d00625af0f4cf0328a335
Sep 28 08:33:27 dist-test-slave-qp2n krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584407, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:33:27.846340 28449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:27.800788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:40693 (local address 127.15.134.254:34443)
0928 08:33:27.826345 (+ 25557us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:27.826349 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:27.826367 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:33:27.826426 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:27.826430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:27.826537 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:27.826655 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:27.829069 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:27.829866 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:27.832302 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:27.832952 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:27.833081 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:27.837752 (+ 4671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:27.837760 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:27.837766 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:27.837809 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:27.842183 (+ 4374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:27.843316 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:27.843322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:27.843325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:27.843492 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:27.844037 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:27.844041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:27.844044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:27.844375 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:27.844507 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:27.845551 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0928 08:33:27.845814 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25431,"thread_start_us":25316,"threads_started":1}
I20260928 08:33:27.847255 28450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:27.801401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:34443 (local address 127.15.134.193:40693)
0928 08:33:27.819727 (+ 18326us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:27.819804 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:27.823124 (+ 3320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:27.826974 (+ 3850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:27.826987 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:27.827807 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:33:27.828709 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:27.828737 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:27.830082 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:27.830088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:27.832109 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:27.832127 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:27.833416 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:27.833428 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:27.833737 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:27.834621 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:33:27.834652 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:33:27.837472 (+ 2820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:33:27.842463 (+ 4991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:27.842475 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:27.842496 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:27.843114 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:27.843626 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:27.843631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:27.843634 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:27.843911 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:27.844586 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:33:27.844594 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:33:27.845074 (+ 480us) client_negotiation.cc:769] Sending connection context
0928 08:33:27.845381 (+ 307us) client_negotiation.cc:240] Negotiation successful
0928 08:33:27.845930 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17853,"thread_start_us":25631,"threads_started":1}
I20260928 08:33:27.849236 28447 heartbeater.cc:347] Connected to a master server at 127.15.134.254:34443
I20260928 08:33:27.849591 28447 heartbeater.cc:464] Registering TS with master...
I20260928 08:33:27.850598 28447 heartbeater.cc:511] Master 127.15.134.254:34443 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.15.134.194@KRBTEST.COM; defaulting to no policy
I20260928 08:33:27.853411 28233 ts_manager.cc:194] Registered new tserver with Master: eaaaa27d00d3443faf6808a5cfe6d7ee (127.15.134.193:37011)
I20260928 08:33:27.855225 28233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.193@KRBTEST.COM'} at 127.15.134.193:40693
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
I20260928 08:33:27.897600 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.134.212:35389/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:0
--local_ip_for_outbound_sockets=127.15.134.194
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:34443
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:33:28 dist-test-slave-qp2n krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584407, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for HTTP/127.15.134.212@KRBTEST.COM
W20260928 08:33:28.378402 28453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:33:28.378741 28453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:33:28.378906 28453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:33:28.386444 28453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:33:28.386586 28453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:33:28.386765 28453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:33:28.396929 28453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.15.134.212:35389/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:34443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:33:28.398587 28453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:33:28.400046 28453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:33:28.414357 28470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:28.414337 28468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:28.414932 28467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:28.415212 28453 server_base.cc:1034] running on GCE node
I20260928 08:33:28.416036 28453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:33:28.417471 28453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:33:28.418766 28453 hybrid_clock.cc:648] HybridClock initialized: now 1790584408418691 us; error 60 us; skew 500 ppm
I20260928 08:33:28.419166 28453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:33:28 dist-test-slave-qp2n krb5kdc[27757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584408, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:33:28.427475 28453 init.cc:377] Logged in from keytab as kudu/127.15.134.194@KRBTEST.COM (short username kudu)
I20260928 08:33:28.429371 28453 webserver.cc:460] Webserver started at http://127.15.134.194:45487/ using document root <none> and password file <none>
I20260928 08:33:28.430295 28453 fs_manager.cc:359] Metadata directory not provided
I20260928 08:33:28.430441 28453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:33:28.430792 28453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:33:28 dist-test-slave-qp2n krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584408, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for HTTP/127.15.134.212@KRBTEST.COM
I20260928 08:33:28.449100 28453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/data/instance:
uuid: "d73d7d3e291844639d88b7ce89793aaa"
format_stamp: "Formatted at 2026-09-28 08:33:28 on dist-test-slave-qp2n"
server_key: "3282c0d300e31c61d5ea1df7a5a41ea2"
server_key_iv: "f51ea090d49955ab063413a7604984dc"
server_key_version: "kuduclusterkey@0"
I20260928 08:33:28.450294 28453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance:
uuid: "d73d7d3e291844639d88b7ce89793aaa"
format_stamp: "Formatted at 2026-09-28 08:33:28 on dist-test-slave-qp2n"
server_key: "3282c0d300e31c61d5ea1df7a5a41ea2"
server_key_iv: "f51ea090d49955ab063413a7604984dc"
server_key_version: "kuduclusterkey@0"
I20260928 08:33:28.457913 28453 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260928 08:33:28.475715 28477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:28.477658 28453 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 08:33:28.477860 28453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "d73d7d3e291844639d88b7ce89793aaa"
format_stamp: "Formatted at 2026-09-28 08:33:28 on dist-test-slave-qp2n"
server_key: "3282c0d300e31c61d5ea1df7a5a41ea2"
server_key_iv: "f51ea090d49955ab063413a7604984dc"
server_key_version: "kuduclusterkey@0"
I20260928 08:33:28.478140 28453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:33:28.511554 28453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:33:28.516999 28453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:33:28.517379 28453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:33:28.518710 28453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:33:28.520648 28453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:33:28.520748 28453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260928 08:33:28.520881 28453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:33:28.520936 28453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:28.559751 28453 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:35583
I20260928 08:33:28.559754 28589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:35583 every 8 connection(s)
I20260928 08:33:28.561573 28453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:33:28.563262 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 28453
I20260928 08:33:28.563437 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance
I20260928 08:33:28.563963 15899 external_mini_cluster.cc:1428] Setting key 18a8eaf92ac9364bffc037dd8f8e3488
Sep 28 08:33:28 dist-test-slave-qp2n krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584408, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.134.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.195@KRBTEST.COM" created.
I20260928 08:33:28.595194 28592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:28.564667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:46991 (local address 127.15.134.254:34443)
0928 08:33:28.577064 (+ 12397us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:28.577068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:28.577086 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:33:28.577139 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:28.577142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:28.577218 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:28.577316 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:28.579556 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:28.580428 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:28.583198 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:28.583719 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:28.583939 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:28.588489 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:28.588494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:28.588498 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:28.588534 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:28.591492 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:28.592325 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:28.592331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:28.592334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:28.592474 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:28.592929 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:28.592932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:28.592934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:28.593216 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:28.593327 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:28.594382 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0928 08:33:28.594667 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12280,"thread_start_us":12157,"threads_started":1}
I20260928 08:33:28.595732 28593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:28.565392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:34443 (local address 127.15.134.194:46991)
0928 08:33:28.573541 (+ 8149us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:28.573611 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:28.575934 (+ 2323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:28.577655 (+ 1721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:28.577671 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:28.578355 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:33:28.579310 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:28.579341 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:28.580606 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:28.580649 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:28.583026 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:28.583047 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:28.583992 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:28.584003 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:28.584359 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:28.585246 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:33:28.585289 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:33:28.588211 (+ 2922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:33:28.591641 (+ 3430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:28.591649 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:28.591665 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:28.592200 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:28.592572 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:28.592575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:28.592577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:28.592819 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:28.593325 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:33:28.593331 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:33:28.593672 (+ 341us) client_negotiation.cc:769] Sending connection context
0928 08:33:28.594045 (+ 373us) client_negotiation.cc:240] Negotiation successful
0928 08:33:28.594451 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7734,"thread_start_us":7455,"threads_started":1}
I20260928 08:33:28.597461 28590 heartbeater.cc:347] Connected to a master server at 127.15.134.254:34443
I20260928 08:33:28.597784 28590 heartbeater.cc:464] Registering TS with master...
I20260928 08:33:28.598522 28590 heartbeater.cc:511] Master 127.15.134.254:34443 requested a full tablet report, sending...
I20260928 08:33:28.600107 28233 ts_manager.cc:194] Registered new tserver with Master: d73d7d3e291844639d88b7ce89793aaa (127.15.134.194:35583)
I20260928 08:33:28.601012 28233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.194@KRBTEST.COM'} at 127.15.134.194:46991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
I20260928 08:33:28.624274 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.134.212:35389/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:0
--local_ip_for_outbound_sockets=127.15.134.195
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:34443
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:33:28.859591 28447 heartbeater.cc:503] Master 127.15.134.254:34443 was elected leader, sending a full tablet report...
W20260928 08:33:28.959975 28596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:33:28.960304 28596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:33:28.960479 28596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:33:28.967902 28596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:33:28.968075 28596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:33:28.968271 28596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:33:28.978622 28596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.15.134.212:35389/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:34443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:33:28.980043 28596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:33:28.981482 28596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:33:28.995524 28602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:28.995563 28604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:28.995585 28601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:28.996361 28596 server_base.cc:1034] running on GCE node
I20260928 08:33:28.997097 28596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:33:28.998154 28596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:33:28.999387 28596 hybrid_clock.cc:648] HybridClock initialized: now 1790584408999324 us; error 52 us; skew 500 ppm
I20260928 08:33:28.999722 28596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:33:28 dist-test-slave-qp2n krb5kdc[27757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584408, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:33:29.002916 28596 init.cc:377] Logged in from keytab as kudu/127.15.134.195@KRBTEST.COM (short username kudu)
I20260928 08:33:29.004431 28596 webserver.cc:460] Webserver started at http://127.15.134.195:41291/ using document root <none> and password file <none>
I20260928 08:33:29.005178 28596 fs_manager.cc:359] Metadata directory not provided
I20260928 08:33:29.005342 28596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:33:29.005656 28596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:33:29 dist-test-slave-qp2n krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584408, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for HTTP/127.15.134.212@KRBTEST.COM
I20260928 08:33:29.023348 28596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/data/instance:
uuid: "f0dfd0880b7b45beb6f01a39c8e59c33"
format_stamp: "Formatted at 2026-09-28 08:33:29 on dist-test-slave-qp2n"
server_key: "9498e01ff9135f67a397eec600b469d3"
server_key_iv: "e22dcebf353d1bd9f1295c403939ec6e"
server_key_version: "kuduclusterkey@0"
I20260928 08:33:29.024499 28596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance:
uuid: "f0dfd0880b7b45beb6f01a39c8e59c33"
format_stamp: "Formatted at 2026-09-28 08:33:29 on dist-test-slave-qp2n"
server_key: "9498e01ff9135f67a397eec600b469d3"
server_key_iv: "e22dcebf353d1bd9f1295c403939ec6e"
server_key_version: "kuduclusterkey@0"
I20260928 08:33:29.031600 28596 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260928 08:33:29.049181 28611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:29.051100 28596 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 08:33:29.051306 28596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "f0dfd0880b7b45beb6f01a39c8e59c33"
format_stamp: "Formatted at 2026-09-28 08:33:29 on dist-test-slave-qp2n"
server_key: "9498e01ff9135f67a397eec600b469d3"
server_key_iv: "e22dcebf353d1bd9f1295c403939ec6e"
server_key_version: "kuduclusterkey@0"
I20260928 08:33:29.051582 28596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:33:29.078886 28596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:33:29.082798 28596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:33:29.083069 28596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:33:29.084252 28596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:33:29.085857 28596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:33:29.085932 28596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:29.086024 28596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:33:29.086071 28596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:29.121362 28596 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:42823
I20260928 08:33:29.121411 28723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:42823 every 8 connection(s)
I20260928 08:33:29.122691 28596 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:33:29.128053 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 28596
I20260928 08:33:29.128248 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance
I20260928 08:33:29.128783 15899 external_mini_cluster.cc:1428] Setting key beb2ca35d339754d89bdc4ec2a9e43f9
Sep 28 08:33:29 dist-test-slave-qp2n krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584408, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:33:29.143625 28726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:29.125277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:35421 (local address 127.15.134.254:34443)
0928 08:33:29.125734 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:29.125738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:29.126843 (+ 1105us) server_negotiation.cc:407] Connection header received
0928 08:33:29.128886 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:29.128892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:29.128987 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:29.129119 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:29.130758 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:29.131867 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:29.133513 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:29.133923 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:29.134017 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:29.137346 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:29.137351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:29.137355 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:29.137387 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:29.140100 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:29.140931 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:29.140935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:29.140937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:29.141080 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:29.141460 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:29.141463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:29.141465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:29.141741 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:29.141853 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:29.142857 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0928 08:33:29.143214 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":223,"threads_started":1}
I20260928 08:33:29.144578 28727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:29.125898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:34443 (local address 127.15.134.195:35421)
0928 08:33:29.126684 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:29.126727 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:29.128576 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:29.129351 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:29.129359 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:29.129889 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:33:29.130559 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:29.130576 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:29.132039 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:29.132046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:29.133379 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:29.133391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:29.134115 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:29.134120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:29.134314 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:29.135048 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:33:29.135075 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:33:29.137132 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:33:29.140240 (+ 3108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:29.140248 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:29.140263 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:29.140804 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:29.141176 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:29.141179 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:29.141182 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:29.141363 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:29.141844 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:33:29.141850 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:33:29.142214 (+ 364us) client_negotiation.cc:769] Sending connection context
0928 08:33:29.142555 (+ 341us) client_negotiation.cc:240] Negotiation successful
0928 08:33:29.143123 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":364,"threads_started":1}
I20260928 08:33:29.145954 28724 heartbeater.cc:347] Connected to a master server at 127.15.134.254:34443
I20260928 08:33:29.146268 28724 heartbeater.cc:464] Registering TS with master...
I20260928 08:33:29.147006 28724 heartbeater.cc:511] Master 127.15.134.254:34443 requested a full tablet report, sending...
I20260928 08:33:29.148368 28233 ts_manager.cc:194] Registered new tserver with Master: f0dfd0880b7b45beb6f01a39c8e59c33 (127.15.134.195:42823)
I20260928 08:33:29.149279 28233 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.195@KRBTEST.COM'} at 127.15.134.195:35421
I20260928 08:33:29.157373 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:33:29.159186 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 28314
Sep 28 08:33:29 dist-test-slave-qp2n krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.193@KRBTEST.COM
I20260928 08:33:29.180091 28729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:29.160149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51232 (local address 127.15.134.193:37011)
0928 08:33:29.165134 (+ 4985us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:29.165143 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:29.165176 (+ 33us) server_negotiation.cc:407] Connection header received
0928 08:33:29.165241 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:29.165250 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:29.165503 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:29.165653 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:29.167105 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:29.168148 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:29.169877 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:29.170445 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:29.170557 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:29.174729 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:29.174741 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:29.174755 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:29.174793 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:29.177620 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:29.178257 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:29.178264 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:29.178266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:29.178414 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:29.178779 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:29.178783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:29.178786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:29.179132 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:29.179372 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:29.179617 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 08:33:29.179815 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4818,"thread_start_us":4654,"threads_started":1}
I20260928 08:33:29.181373 28442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:33:29.265719 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 28314
I20260928 08:33:29.282867 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 28453
Sep 28 08:33:29 dist-test-slave-qp2n krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.194@KRBTEST.COM
I20260928 08:33:29.299235 28731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:29.283892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45002 (local address 127.15.134.194:35583)
0928 08:33:29.284460 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:29.284467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:29.284494 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:33:29.284555 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:29.284561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:29.284852 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:29.285016 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:29.286139 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:29.287124 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:29.288452 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:29.289111 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:29.289214 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:29.293594 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:29.293608 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:29.293624 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:29.293661 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:29.296491 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:29.297228 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:29.297235 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:29.297237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:29.297396 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:29.297773 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:29.297778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:29.297780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:29.298134 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:29.298415 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:29.298604 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 08:33:29.298799 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":301,"threads_started":1}
I20260928 08:33:29.300505 28585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:33:29.386121 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 28453
I20260928 08:33:29.407018 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 28596
Sep 28 08:33:29 dist-test-slave-qp2n krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.195@KRBTEST.COM
I20260928 08:33:29.423485 28733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:29.407968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44298 (local address 127.15.134.195:42823)
0928 08:33:29.408666 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:29.408677 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:29.408718 (+ 41us) server_negotiation.cc:407] Connection header received
0928 08:33:29.408781 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:29.408788 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:29.409037 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:29.409161 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:29.410672 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:29.411671 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:29.413071 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:29.413610 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:29.413705 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:29.418085 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:29.418096 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:29.418111 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:29.418143 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:29.421123 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:29.421721 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:29.421728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:29.421730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:29.421905 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:29.422214 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:29.422217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:29.422219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:29.422533 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:29.422801 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:29.423015 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 08:33:29.423196 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":318,"threads_started":1}
I20260928 08:33:29.424817 28719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:33:29.507863 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 28596
I20260928 08:33:29.526041 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 28201
I20260928 08:33:29.527485 28264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:33:29.654026 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 28201
I20260928 08:33:29.679932 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:34443
--webserver_interface=127.15.134.254
--webserver_port=43051
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:34443
--ranger_config_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.134.212:35389/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.134.212:35389/kms with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:33:30.014154 28736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:33:30.014462 28736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:33:30.014568 28736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:33:30.022336 28736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:33:30.022468 28736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:33:30.022516 28736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:33:30.022564 28736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:33:30.022615 28736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:33:30.033047 28736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.134.212:35389/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:34443
--ranger_config_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:34443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=43051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:33:30.034389 28736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:33:30.035720 28736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:33:30.048506 28744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:30.048676 28741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:30.048593 28742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:30.049315 28736 server_base.cc:1034] running on GCE node
I20260928 08:33:30.050199 28736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:33:30.051718 28736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:33:30.052963 28736 hybrid_clock.cc:648] HybridClock initialized: now 1790584410052917 us; error 34 us; skew 500 ppm
I20260928 08:33:30.053354 28736 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:33:30 dist-test-slave-qp2n krb5kdc[27757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584410, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:33:30.056907 28736 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:33:30.058373 28736 webserver.cc:460] Webserver started at http://127.15.134.254:43051/ using document root <none> and password file <none>
I20260928 08:33:30.059090 28736 fs_manager.cc:359] Metadata directory not provided
I20260928 08:33:30.059233 28736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 08:33:30 dist-test-slave-qp2n krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584410, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for HTTP/127.15.134.212@KRBTEST.COM
I20260928 08:33:30.109769 28736 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:33:30.113631 28752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:30.115389 28736 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260928 08:33:30.115593 28736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "9e343dcafe71485180a75c406a699a88"
format_stamp: "Formatted at 2026-09-28 08:33:26 on dist-test-slave-qp2n"
server_key: "379c41b05001b6e6e894847083fe0c72"
server_key_iv: "ec7bf14fc4f51a6b3aaae06443a3ed2b"
server_key_version: "kuduclusterkey@0"
I20260928 08:33:30.116917 28736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:33:30.142443 28736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:33:30.147137 28736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:33:30.147472 28736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:33:30.169241 28736 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:34443
I20260928 08:33:30.169261 28803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:34443 every 8 connection(s)
I20260928 08:33:30.170967 28736 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:33:30.174273 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 28736
I20260928 08:33:30.175165 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.134.212:35389/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:37011
--local_ip_for_outbound_sockets=127.15.134.193
--tserver_master_addrs=127.15.134.254:34443
--webserver_port=41005
--webserver_interface=127.15.134.193
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.134.212:35389/kms with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:33:30.179018 28804 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:33:30.182799 28804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88: Bootstrap starting.
I20260928 08:33:30.203807 28804 log.cc:824] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88: Log is configured to *not* fsync() on all Append() calls
I20260928 08:33:30.213516 28804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:33:30.214165 28804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88: Bootstrap complete.
I20260928 08:33:30.225030 28804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e343dcafe71485180a75c406a699a88" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 34443 } }
I20260928 08:33:30.226006 28804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e343dcafe71485180a75c406a699a88, State: Initialized, Role: FOLLOWER
I20260928 08:33:30.226759 28804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e343dcafe71485180a75c406a699a88" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 34443 } }
I20260928 08:33:30.227016 28804 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:33:30.227161 28804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:33:30.227376 28804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [term 1 FOLLOWER]: Advancing to term 2
I20260928 08:33:30.231673 28804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e343dcafe71485180a75c406a699a88" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 34443 } }
I20260928 08:33:30.232445 28804 leader_election.cc:302] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e343dcafe71485180a75c406a699a88; no voters:
I20260928 08:33:30.233223 28804 leader_election.cc:288] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 08:33:30.233403 28808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 08:33:30.234488 28808 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [term 2 LEADER]: Becoming Leader. State: Replica: 9e343dcafe71485180a75c406a699a88, State: Running, Role: LEADER
I20260928 08:33:30.235183 28808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e343dcafe71485180a75c406a699a88" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 34443 } }
I20260928 08:33:30.235597 28804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:33:30.237804 28810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e343dcafe71485180a75c406a699a88. Latest consensus state: current_term: 2 leader_uuid: "9e343dcafe71485180a75c406a699a88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e343dcafe71485180a75c406a699a88" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 34443 } } }
I20260928 08:33:30.238160 28810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [sys.catalog]: This master's current role is: LEADER
I20260928 08:33:30.238685 28809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9e343dcafe71485180a75c406a699a88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e343dcafe71485180a75c406a699a88" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 34443 } } }
I20260928 08:33:30.238963 28809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88 [sys.catalog]: This master's current role is: LEADER
I20260928 08:33:30.244023 28814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:33:30.250686 28814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:33:30.252354 28814 catalog_manager.cc:1259] Loaded cluster ID: 9447a648bf9c48bda9211f285da479c1
I20260928 08:33:30.252544 28814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:33:30.256882 28814 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:33:30.258452 28814 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88: Loaded TSK: 0
I20260928 08:33:30.260087 28814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:33:30.266607 28804 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260928 08:33:30.605322 28806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:33:30.605860 28806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:33:30.606108 28806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:33:30.618221 28806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:33:30.618460 28806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:33:30.618719 28806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:33:30.634855 28806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.134.212:35389/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:37011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=41005
--tserver_master_addrs=127.15.134.254:34443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:33:30.636953 28806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:33:30.638783 28806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:33:30.658231 28842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:30.661124 28843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:30.669541 28845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:30.672991 28806 server_base.cc:1034] running on GCE node
I20260928 08:33:30.674072 28806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:33:30.682425 28806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:33:30.683807 28806 hybrid_clock.cc:648] HybridClock initialized: now 1790584410683729 us; error 80 us; skew 500 ppm
I20260928 08:33:30.684322 28806 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:33:30 dist-test-slave-qp2n krb5kdc[27757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584410, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:33:30.689416 28806 init.cc:377] Logged in from keytab as kudu/127.15.134.193@KRBTEST.COM (short username kudu)
I20260928 08:33:30.691867 28806 webserver.cc:460] Webserver started at http://127.15.134.193:41005/ using document root <none> and password file <none>
I20260928 08:33:30.692978 28806 fs_manager.cc:359] Metadata directory not provided
I20260928 08:33:30.693171 28806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 08:33:30 dist-test-slave-qp2n krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584410, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for HTTP/127.15.134.212@KRBTEST.COM
I20260928 08:33:30.735594 28806 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.007s sys 0.003s
I20260928 08:33:30.752030 28853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:30.754983 28806 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.010s sys 0.004s
I20260928 08:33:30.755290 28806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "eaaaa27d00d3443faf6808a5cfe6d7ee"
format_stamp: "Formatted at 2026-09-28 08:33:27 on dist-test-slave-qp2n"
server_key: "9eaf1eb28c472a4870dadee52902891f"
server_key_iv: "aa0a2482e510ba165df282371c889465"
server_key_version: "kuduclusterkey@0"
I20260928 08:33:30.757020 28806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:33:30.814236 28806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:33:30.819330 28806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:33:30.819684 28806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:33:30.824656 28806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:33:30.826256 28806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:33:30.826395 28806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:30.826535 28806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:33:30.826618 28806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:30.892187 28806 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:37011
I20260928 08:33:30.892270 28966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:37011 every 8 connection(s)
I20260928 08:33:30.894001 28806 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:33:30.895412 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 28806
I20260928 08:33:30.896245 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.134.212:35389/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:35583
--local_ip_for_outbound_sockets=127.15.134.194
--tserver_master_addrs=127.15.134.254:34443
--webserver_port=45487
--webserver_interface=127.15.134.194
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.134.212:35389/kms with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:33:30 dist-test-slave-qp2n krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584410, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:33:30.941809 28970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:30.901757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:46651 (local address 127.15.134.254:34443)
0928 08:33:30.907961 (+ 6204us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:30.907982 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:30.911690 (+ 3708us) server_negotiation.cc:407] Connection header received
0928 08:33:30.912158 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:30.912175 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:30.912694 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:30.913083 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:30.915342 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:30.917591 (+ 2249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:30.920100 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:30.926079 (+ 5979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:30.926348 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:30.931713 (+ 5365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:30.931722 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:30.931740 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:30.931782 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:30.935751 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:30.937079 (+ 1328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:30.937089 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:30.937099 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:30.937329 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:30.937968 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:30.937973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:30.937977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:30.938695 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:30.938930 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:30.939947 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0928 08:33:30.940446 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5878,"thread_start_us":5651,"threads_started":1}
I20260928 08:33:30.941815 28969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:30.897682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:34443 (local address 127.15.134.193:46651)
0928 08:33:30.908706 (+ 11024us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:30.908769 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:30.911260 (+ 2491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:30.913737 (+ 2477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:30.913747 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:30.914351 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:33:30.915083 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:30.915106 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:30.917808 (+ 2702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:30.917817 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:30.919899 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:30.919918 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:30.927547 (+ 7629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:30.927556 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:30.927873 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:30.928749 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:33:30.928783 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:33:30.931384 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:33:30.936041 (+ 4657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:30.936054 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:30.936072 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:30.936815 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:30.937509 (+ 694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:30.937515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:30.937518 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:30.937789 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:30.938892 (+ 1103us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:33:30.938904 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:33:30.939346 (+ 442us) client_negotiation.cc:769] Sending connection context
0928 08:33:30.939686 (+ 340us) client_negotiation.cc:240] Negotiation successful
0928 08:33:30.940369 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10598,"thread_start_us":10361,"threads_started":1}
I20260928 08:33:30.946379 28967 heartbeater.cc:347] Connected to a master server at 127.15.134.254:34443
I20260928 08:33:30.946717 28967 heartbeater.cc:464] Registering TS with master...
I20260928 08:33:30.947594 28967 heartbeater.cc:511] Master 127.15.134.254:34443 requested a full tablet report, sending...
W20260928 08:33:30.950798 28967 heartbeater.cc:644] Failed to heartbeat to 127.15.134.254:34443 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 08:33:30.951028 28967 heartbeater.cc:464] Registering TS with master...
I20260928 08:33:30.951304 28967 heartbeater.cc:511] Master 127.15.134.254:34443 requested a full tablet report, sending...
I20260928 08:33:30.952582 28967 heartbeater.cc:464] Registering TS with master...
I20260928 08:33:30.952952 28967 heartbeater.cc:511] Master 127.15.134.254:34443 requested a full tablet report, sending...
W20260928 08:33:30.954021 28967 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:33:31.319949 28971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:33:31.320278 28971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:33:31.320447 28971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:33:31.330049 28971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:33:31.330191 28971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:33:31.330444 28971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:33:31.340809 28971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.134.212:35389/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:35583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=45487
--tserver_master_addrs=127.15.134.254:34443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:33:31.342183 28971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:33:31.343562 28971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:33:31.361498 28985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:31.362298 28986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:31.365872 28971 server_base.cc:1034] running on GCE node
W20260928 08:33:31.366139 28988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:31.366869 28971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:33:31.367921 28971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:33:31.369171 28971 hybrid_clock.cc:648] HybridClock initialized: now 1790584411369109 us; error 54 us; skew 500 ppm
I20260928 08:33:31.369508 28971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:33:31 dist-test-slave-qp2n krb5kdc[27757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584411, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:33:31.372859 28971 init.cc:377] Logged in from keytab as kudu/127.15.134.194@KRBTEST.COM (short username kudu)
I20260928 08:33:31.376196 28971 webserver.cc:460] Webserver started at http://127.15.134.194:45487/ using document root <none> and password file <none>
I20260928 08:33:31.377105 28971 fs_manager.cc:359] Metadata directory not provided
I20260928 08:33:31.377260 28971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 08:33:31 dist-test-slave-qp2n krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584411, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for HTTP/127.15.134.212@KRBTEST.COM
I20260928 08:33:31.413317 28971 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.006s sys 0.005s
I20260928 08:33:31.423822 28996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:31.426307 28971 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.004s
I20260928 08:33:31.426632 28971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "d73d7d3e291844639d88b7ce89793aaa"
format_stamp: "Formatted at 2026-09-28 08:33:28 on dist-test-slave-qp2n"
server_key: "3282c0d300e31c61d5ea1df7a5a41ea2"
server_key_iv: "f51ea090d49955ab063413a7604984dc"
server_key_version: "kuduclusterkey@0"
I20260928 08:33:31.428333 28971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:33:31.466557 28971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:33:31.484200 28971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:33:31.484653 28971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:33:31.499576 28971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:33:31.516657 28971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:33:31.516858 28971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260928 08:33:31.517022 28971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:33:31.517129 28971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:33:31 dist-test-slave-qp2n krb5kdc[27757](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790584411, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:33:31.824211 29111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:35583 every 8 connection(s)
I20260928 08:33:31.824738 28971 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:35583
I20260928 08:33:31.826828 28971 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:33:31.835451 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 28971
I20260928 08:33:31.836264 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.15.134.212:35389/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:42823
--local_ip_for_outbound_sockets=127.15.134.195
--tserver_master_addrs=127.15.134.254:34443
--webserver_port=41291
--webserver_interface=127.15.134.195
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.15.134.212:35389/kms with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:33:31 dist-test-slave-qp2n krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584411, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:33:31.910629 29114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:31.836810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:37683 (local address 127.15.134.254:34443)
0928 08:33:31.870621 (+ 33811us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:31.870628 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:31.874805 (+ 4177us) server_negotiation.cc:407] Connection header received
0928 08:33:31.874889 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:31.874897 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:31.875001 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:31.875107 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:31.878597 (+ 3490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:31.879752 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:31.882381 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:31.883225 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:31.883334 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:31.901997 (+ 18663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:31.902008 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:31.902014 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:31.902074 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:31.906210 (+ 4136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:31.907434 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:31.907442 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:31.907449 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:31.907608 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:31.908188 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:31.908196 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:31.908202 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:31.908684 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:31.908846 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:31.910040 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0928 08:33:31.910275 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31686,"thread_start_us":31906,"threads_started":1}
I20260928 08:33:31.911818 29113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:31.834438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:34443 (local address 127.15.134.194:37683)
0928 08:33:31.871305 (+ 36867us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:31.871369 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:31.874265 (+ 2896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:31.876425 (+ 2160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:31.876442 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:31.877479 (+ 1037us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:33:31.878364 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:31.878393 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:31.879999 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:31.880005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:31.882166 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:31.882183 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:31.883708 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:31.883716 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:31.884146 (+ 430us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:31.886164 (+ 2018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:33:31.886202 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:33:31.901595 (+ 15393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:33:31.906457 (+ 4862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:31.906470 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:31.906490 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:31.907240 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:31.907759 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:31.907766 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:31.907772 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:31.908034 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:31.908939 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:33:31.908948 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:33:31.909474 (+ 526us) client_negotiation.cc:769] Sending connection context
0928 08:33:31.909835 (+ 361us) client_negotiation.cc:240] Negotiation successful
0928 08:33:31.910421 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36507,"thread_start_us":40716,"threads_started":1}
I20260928 08:33:31.913468 29112 heartbeater.cc:347] Connected to a master server at 127.15.134.254:34443
I20260928 08:33:31.913793 29112 heartbeater.cc:464] Registering TS with master...
I20260928 08:33:31.914634 29112 heartbeater.cc:511] Master 127.15.134.254:34443 requested a full tablet report, sending...
I20260928 08:33:31.916747 28769 ts_manager.cc:194] Registered new tserver with Master: d73d7d3e291844639d88b7ce89793aaa (127.15.134.194:35583)
I20260928 08:33:31.918843 28769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.194@KRBTEST.COM'} at 127.15.134.194:37683
I20260928 08:33:31.954181 28967 heartbeater.cc:464] Registering TS with master...
I20260928 08:33:31.954665 28967 heartbeater.cc:511] Master 127.15.134.254:34443 requested a full tablet report, sending...
I20260928 08:33:31.955973 28769 ts_manager.cc:194] Registered new tserver with Master: eaaaa27d00d3443faf6808a5cfe6d7ee (127.15.134.193:37011)
I20260928 08:33:31.957201 28769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.193@KRBTEST.COM'} at 127.15.134.193:46651
Sep 28 08:33:32 dist-test-slave-qp2n krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584411, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for HTTP/127.15.134.212@KRBTEST.COM
W20260928 08:33:32.323076 29115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:33:32.323429 29115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:33:32.323580 29115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:33:32.331166 29115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:33:32.331324 29115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:33:32.331542 29115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:33:32.345070 29115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.15.134.212:35389/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:42823
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=41291
--tserver_master_addrs=127.15.134.254:34443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:33:32.346879 29115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:33:32.348805 29115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:33:32.371163 29123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:32.371174 29122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:32.382757 29115 server_base.cc:1034] running on GCE node
W20260928 08:33:32.383149 29126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:32.383951 29115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
Sep 28 08:33:32 dist-test-slave-qp2n krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584411, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for HTTP/127.15.134.212@KRBTEST.COM
I20260928 08:33:32.388001 29115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:33:32.389856 29115 hybrid_clock.cc:648] HybridClock initialized: now 1790584412389778 us; error 66 us; skew 500 ppm
I20260928 08:33:32.390262 29115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:33:32 dist-test-slave-qp2n krb5kdc[27757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584412, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:33:32.396056 29115 init.cc:377] Logged in from keytab as kudu/127.15.134.195@KRBTEST.COM (short username kudu)
I20260928 08:33:32.403844 29115 webserver.cc:460] Webserver started at http://127.15.134.195:41291/ using document root <none> and password file <none>
I20260928 08:33:32.405300 29115 fs_manager.cc:359] Metadata directory not provided
I20260928 08:33:32.405509 29115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 08:33:32 dist-test-slave-qp2n krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584412, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for HTTP/127.15.134.212@KRBTEST.COM
I20260928 08:33:32.486606 29115 fs_manager.cc:709] Time spent opening directory manager: real 0.020s user 0.004s sys 0.000s
I20260928 08:33:32.497257 29133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:32.499254 29115 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.009s sys 0.001s
I20260928 08:33:32.499871 29115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "f0dfd0880b7b45beb6f01a39c8e59c33"
format_stamp: "Formatted at 2026-09-28 08:33:29 on dist-test-slave-qp2n"
server_key: "9498e01ff9135f67a397eec600b469d3"
server_key_iv: "e22dcebf353d1bd9f1295c403939ec6e"
server_key_version: "kuduclusterkey@0"
I20260928 08:33:32.501302 29115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:33:32.559206 29115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:33:32.565179 29115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:33:32.565567 29115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:33:32.566876 29115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:33:32.575049 29115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:33:32.575263 29115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 08:33:32.575431 29115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:33:32.575559 29115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:32.669410 29115 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:42823
I20260928 08:33:32.669512 29252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:42823 every 8 connection(s)
I20260928 08:33:32.671053 29115 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:33:32.673417 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 29115
I20260928 08:33:32.705084 29257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:32.674637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50964 (local address 127.15.134.254:34443)
0928 08:33:32.686016 (+ 11379us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:32.686025 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:32.686054 (+ 29us) server_negotiation.cc:407] Connection header received
0928 08:33:32.686158 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:32.686167 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:32.686276 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:32.689246 (+ 2970us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:32.689515 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:32.691070 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:32.693693 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:32.694386 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:32.695314 (+ 928us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:32.698210 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:32.698218 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:32.698221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:32.698262 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:32.701773 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:32.702746 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:32.702751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:32.702755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:32.702968 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:32.703439 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:32.703443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:32.703446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:32.703695 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:32.703980 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:32.704372 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 08:33:32.704658 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11218,"thread_start_us":16731,"threads_started":1}
Sep 28 08:33:32 dist-test-slave-qp2n krb5kdc[27757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584412, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:33:32.713250 29256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:32.675049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:48649 (local address 127.15.134.254:34443)
0928 08:33:32.691689 (+ 16640us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:32.691697 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:32.693588 (+ 1891us) server_negotiation.cc:407] Connection header received
0928 08:33:32.695184 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:32.695192 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:32.695304 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:32.695605 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:32.697952 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:32.699427 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:32.702000 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:32.702390 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:32.702532 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:32.707704 (+ 5172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:32.707711 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:32.707715 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:32.707759 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:32.710437 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:32.711156 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:32.711159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:32.711161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:32.711272 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:32.711689 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:32.711692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:32.711694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:32.711936 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:32.712037 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:32.712797 (+ 760us) server_negotiation.cc:299] Negotiation successful
0928 08:33:32.712971 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16488,"thread_start_us":10690,"threads_started":1}
I20260928 08:33:32.714309 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:32.675634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:34443 (local address 127.15.134.195:48649)
0928 08:33:32.692155 (+ 16521us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:32.692225 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:32.694802 (+ 2577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:32.695977 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:32.695992 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:32.696715 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:33:32.697665 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:32.697694 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:32.699627 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:32.699633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:32.701816 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:32.701843 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:32.702671 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:32.702682 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:32.702898 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:32.704160 (+ 1262us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:33:32.704194 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:33:32.707400 (+ 3206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:33:32.710570 (+ 3170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:32.710577 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:32.710591 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:32.711042 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:32.711398 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:32.711401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:32.711404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:32.711585 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:32.712063 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:33:32.712069 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:33:32.712392 (+ 323us) client_negotiation.cc:769] Sending connection context
0928 08:33:32.712705 (+ 313us) client_negotiation.cc:240] Negotiation successful
0928 08:33:32.713118 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16158,"thread_start_us":15744,"threads_started":1}
I20260928 08:33:32.715758 29253 heartbeater.cc:347] Connected to a master server at 127.15.134.254:34443
I20260928 08:33:32.716048 29253 heartbeater.cc:464] Registering TS with master...
I20260928 08:33:32.716807 29253 heartbeater.cc:511] Master 127.15.134.254:34443 requested a full tablet report, sending...
I20260928 08:33:32.718256 28769 ts_manager.cc:194] Registered new tserver with Master: f0dfd0880b7b45beb6f01a39c8e59c33 (127.15.134.195:42823)
I20260928 08:33:32.719185 28769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.195@KRBTEST.COM'} at 127.15.134.195:48649
I20260928 08:33:32.720849 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:33:32.737756 29256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:32.723908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50980 (local address 127.15.134.254:34443)
0928 08:33:32.724134 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:32.724144 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:32.724566 (+ 422us) server_negotiation.cc:407] Connection header received
0928 08:33:32.724795 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:32.724799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:32.724865 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:32.724951 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:32.726499 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:32.727188 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:32.728747 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:32.729181 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:32.729298 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:32.732123 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:32.732128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:32.732132 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:32.732163 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:32.734966 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:32.735797 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:32.735805 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:32.735808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:32.735957 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:32.736305 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:32.736308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:32.736310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:32.736640 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:32.736885 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:32.737208 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 08:33:32.737372 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:33:32.743016 28769 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:33:32.750953 28769 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:33:32.787106 29270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:32.766676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44308 (local address 127.15.134.195:42823)
0928 08:33:32.771166 (+ 4490us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:32.771180 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:32.771228 (+ 48us) server_negotiation.cc:407] Connection header received
0928 08:33:32.771381 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:32.771395 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:32.771751 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:32.774807 (+ 3056us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:33:32.775314 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:32.777744 (+ 2430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:32.783948 (+ 6204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:32.785670 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:32.785865 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:32.786075 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:32.786504 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 08:33:32.786682 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4264,"thread_start_us":4073,"threads_started":1}
I20260928 08:33:32.787158 29268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:32.766496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:42823 (local address 127.0.0.1:44308)
0928 08:33:32.768117 (+ 1621us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:32.768173 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:32.768429 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:32.772095 (+ 3666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:32.772108 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:32.772158 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:33:32.772903 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:32.772925 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:32.780724 (+ 7799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:32.780731 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:32.783702 (+ 2971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:32.783717 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:32.785952 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:32.785959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:32.786206 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:32.786283 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 08:33:32.786444 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 08:33:32.786670 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1236,"mutex_wait_us":63,"thread_start_us":13805,"threads_started":1}
I20260928 08:33:32.788898 29272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:32.766531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:35583 (local address 127.0.0.1:45018)
0928 08:33:32.779677 (+ 13146us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:32.779705 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:32.780567 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:32.781216 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:32.781221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:32.781248 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:33:32.782013 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:32.782032 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:32.783549 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:32.783555 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:32.786016 (+ 2461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:32.786030 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:32.787976 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:32.787983 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:32.788157 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:32.788193 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 08:33:32.788315 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 08:33:32.788462 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12997,"mutex_wait_us":173,"thread_start_us":1026,"threads_started":1}
I20260928 08:33:32.789901 29273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:32.769453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51248 (local address 127.15.134.193:37011)
0928 08:33:32.780293 (+ 10840us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:32.780301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:32.780413 (+ 112us) server_negotiation.cc:407] Connection header received
0928 08:33:32.780712 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:32.780722 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:32.781045 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:32.781688 (+ 643us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:33:32.782502 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:32.784063 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:32.786721 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:32.788527 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:32.788773 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:32.788942 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:32.789528 (+ 586us) server_negotiation.cc:299] Negotiation successful
0928 08:33:32.789663 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10676,"thread_start_us":10536,"threads_started":1}
I20260928 08:33:32.789912 29269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:32.769108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:37011 (local address 127.0.0.1:51248)
0928 08:33:32.780258 (+ 11150us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:32.780280 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:32.780483 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:32.781570 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:32.781575 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:32.781607 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:33:32.782255 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:32.782271 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:32.784241 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:32.784247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:32.786529 (+ 2282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:32.786543 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:32.789103 (+ 2560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:32.789113 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:32.789292 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:32.789326 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 08:33:32.789442 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 08:33:32.789580 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11014,"thread_start_us":10240,"threads_started":1}
I20260928 08:33:32.790504 29271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:32.767318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45018 (local address 127.15.134.194:35583)
0928 08:33:32.769173 (+ 1855us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:32.769185 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:32.779836 (+ 10651us) server_negotiation.cc:407] Connection header received
0928 08:33:32.780788 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:32.780794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:32.781007 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:32.781231 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:33:32.782255 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:32.783383 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:32.786265 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:32.787748 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:32.787912 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:32.788093 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:32.788372 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 08:33:32.788545 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1677,"thread_start_us":1493,"threads_started":1}
I20260928 08:33:32.792097 29181 tablet_service.cc:1467] Processing CreateTablet for tablet 955e3b00b02449209eddf80a22c50899 (DEFAULT_TABLE table=test-table [id=1c18d5ac998145dfb38ce1350461d528]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:33:32.793735 29181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 955e3b00b02449209eddf80a22c50899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:33:32.793838 29046 tablet_service.cc:1467] Processing CreateTablet for tablet 955e3b00b02449209eddf80a22c50899 (DEFAULT_TABLE table=test-table [id=1c18d5ac998145dfb38ce1350461d528]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:33:32.794919 29046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 955e3b00b02449209eddf80a22c50899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:33:32.795121 28902 tablet_service.cc:1467] Processing CreateTablet for tablet 955e3b00b02449209eddf80a22c50899 (DEFAULT_TABLE table=test-table [id=1c18d5ac998145dfb38ce1350461d528]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:33:32.796470 28902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 955e3b00b02449209eddf80a22c50899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:33:32.815553 29274 tablet_bootstrap.cc:492] T 955e3b00b02449209eddf80a22c50899 P d73d7d3e291844639d88b7ce89793aaa: Bootstrap starting.
I20260928 08:33:32.816079 29275 tablet_bootstrap.cc:492] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33: Bootstrap starting.
I20260928 08:33:32.820024 29276 tablet_bootstrap.cc:492] T 955e3b00b02449209eddf80a22c50899 P eaaaa27d00d3443faf6808a5cfe6d7ee: Bootstrap starting.
I20260928 08:33:32.821943 29275 tablet_bootstrap.cc:654] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33: Neither blocks nor log segments found. Creating new log.
I20260928 08:33:32.822096 29274 tablet_bootstrap.cc:654] T 955e3b00b02449209eddf80a22c50899 P d73d7d3e291844639d88b7ce89793aaa: Neither blocks nor log segments found. Creating new log.
I20260928 08:33:32.823176 29275 log.cc:824] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33: Log is configured to *not* fsync() on all Append() calls
I20260928 08:33:32.823508 29274 log.cc:824] T 955e3b00b02449209eddf80a22c50899 P d73d7d3e291844639d88b7ce89793aaa: Log is configured to *not* fsync() on all Append() calls
I20260928 08:33:32.823917 29276 tablet_bootstrap.cc:654] T 955e3b00b02449209eddf80a22c50899 P eaaaa27d00d3443faf6808a5cfe6d7ee: Neither blocks nor log segments found. Creating new log.
I20260928 08:33:32.825196 29276 log.cc:824] T 955e3b00b02449209eddf80a22c50899 P eaaaa27d00d3443faf6808a5cfe6d7ee: Log is configured to *not* fsync() on all Append() calls
I20260928 08:33:32.826601 29275 tablet_bootstrap.cc:492] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33: No bootstrap required, opened a new log
I20260928 08:33:32.826699 29274 tablet_bootstrap.cc:492] T 955e3b00b02449209eddf80a22c50899 P d73d7d3e291844639d88b7ce89793aaa: No bootstrap required, opened a new log
I20260928 08:33:32.826875 29275 ts_tablet_manager.cc:1397] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260928 08:33:32.826973 29274 ts_tablet_manager.cc:1397] T 955e3b00b02449209eddf80a22c50899 P d73d7d3e291844639d88b7ce89793aaa: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260928 08:33:32.829136 29276 tablet_bootstrap.cc:492] T 955e3b00b02449209eddf80a22c50899 P eaaaa27d00d3443faf6808a5cfe6d7ee: No bootstrap required, opened a new log
I20260928 08:33:32.829396 29276 ts_tablet_manager.cc:1397] T 955e3b00b02449209eddf80a22c50899 P eaaaa27d00d3443faf6808a5cfe6d7ee: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260928 08:33:32.834385 29274 raft_consensus.cc:348] T 955e3b00b02449209eddf80a22c50899 P d73d7d3e291844639d88b7ce89793aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dfd0880b7b45beb6f01a39c8e59c33" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 42823 } } peers { permanent_uuid: "d73d7d3e291844639d88b7ce89793aaa" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35583 } } peers { permanent_uuid: "eaaaa27d00d3443faf6808a5cfe6d7ee" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 37011 } }
I20260928 08:33:32.834880 29274 raft_consensus.cc:374] T 955e3b00b02449209eddf80a22c50899 P d73d7d3e291844639d88b7ce89793aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:33:32.834996 29274 raft_consensus.cc:729] T 955e3b00b02449209eddf80a22c50899 P d73d7d3e291844639d88b7ce89793aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73d7d3e291844639d88b7ce89793aaa, State: Initialized, Role: FOLLOWER
I20260928 08:33:32.835997 29274 consensus_queue.cc:260] T 955e3b00b02449209eddf80a22c50899 P d73d7d3e291844639d88b7ce89793aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dfd0880b7b45beb6f01a39c8e59c33" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 42823 } } peers { permanent_uuid: "d73d7d3e291844639d88b7ce89793aaa" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35583 } } peers { permanent_uuid: "eaaaa27d00d3443faf6808a5cfe6d7ee" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 37011 } }
I20260928 08:33:32.837653 29112 heartbeater.cc:503] Master 127.15.134.254:34443 was elected leader, sending a full tablet report...
I20260928 08:33:32.838439 29274 ts_tablet_manager.cc:1428] T 955e3b00b02449209eddf80a22c50899 P d73d7d3e291844639d88b7ce89793aaa: Time spent starting tablet: real 0.011s user 0.000s sys 0.012s
I20260928 08:33:32.838299 29275 raft_consensus.cc:348] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dfd0880b7b45beb6f01a39c8e59c33" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 42823 } } peers { permanent_uuid: "d73d7d3e291844639d88b7ce89793aaa" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35583 } } peers { permanent_uuid: "eaaaa27d00d3443faf6808a5cfe6d7ee" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 37011 } }
I20260928 08:33:32.838773 29275 raft_consensus.cc:374] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:33:32.838861 29275 raft_consensus.cc:729] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0dfd0880b7b45beb6f01a39c8e59c33, State: Initialized, Role: FOLLOWER
I20260928 08:33:32.839543 29275 consensus_queue.cc:260] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dfd0880b7b45beb6f01a39c8e59c33" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 42823 } } peers { permanent_uuid: "d73d7d3e291844639d88b7ce89793aaa" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35583 } } peers { permanent_uuid: "eaaaa27d00d3443faf6808a5cfe6d7ee" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 37011 } }
I20260928 08:33:32.840226 29276 raft_consensus.cc:348] T 955e3b00b02449209eddf80a22c50899 P eaaaa27d00d3443faf6808a5cfe6d7ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dfd0880b7b45beb6f01a39c8e59c33" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 42823 } } peers { permanent_uuid: "d73d7d3e291844639d88b7ce89793aaa" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35583 } } peers { permanent_uuid: "eaaaa27d00d3443faf6808a5cfe6d7ee" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 37011 } }
I20260928 08:33:32.843719 29276 raft_consensus.cc:374] T 955e3b00b02449209eddf80a22c50899 P eaaaa27d00d3443faf6808a5cfe6d7ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:33:32.843839 29276 raft_consensus.cc:729] T 955e3b00b02449209eddf80a22c50899 P eaaaa27d00d3443faf6808a5cfe6d7ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaaaa27d00d3443faf6808a5cfe6d7ee, State: Initialized, Role: FOLLOWER
I20260928 08:33:32.844707 29276 consensus_queue.cc:260] T 955e3b00b02449209eddf80a22c50899 P eaaaa27d00d3443faf6808a5cfe6d7ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dfd0880b7b45beb6f01a39c8e59c33" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 42823 } } peers { permanent_uuid: "d73d7d3e291844639d88b7ce89793aaa" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35583 } } peers { permanent_uuid: "eaaaa27d00d3443faf6808a5cfe6d7ee" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 37011 } }
I20260928 08:33:32.850746 29275 ts_tablet_manager.cc:1428] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33: Time spent starting tablet: real 0.024s user 0.021s sys 0.005s
I20260928 08:33:32.850832 29253 heartbeater.cc:503] Master 127.15.134.254:34443 was elected leader, sending a full tablet report...
I20260928 08:33:32.851347 28967 heartbeater.cc:503] Master 127.15.134.254:34443 was elected leader, sending a full tablet report...
I20260928 08:33:32.854076 29276 ts_tablet_manager.cc:1428] T 955e3b00b02449209eddf80a22c50899 P eaaaa27d00d3443faf6808a5cfe6d7ee: Time spent starting tablet: real 0.025s user 0.016s sys 0.007s
W20260928 08:33:32.869223 29116 tablet.cc:2369] T 955e3b00b02449209eddf80a22c50899 P d73d7d3e291844639d88b7ce89793aaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:33:32.897454 28968 tablet.cc:2369] T 955e3b00b02449209eddf80a22c50899 P eaaaa27d00d3443faf6808a5cfe6d7ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:33:32.923722 29254 tablet.cc:2369] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:33:33.145587 29281 raft_consensus.cc:482] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:33:33.145990 29281 raft_consensus.cc:504] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dfd0880b7b45beb6f01a39c8e59c33" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 42823 } } peers { permanent_uuid: "d73d7d3e291844639d88b7ce89793aaa" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35583 } } peers { permanent_uuid: "eaaaa27d00d3443faf6808a5cfe6d7ee" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 37011 } }
I20260928 08:33:33.147789 29281 leader_election.cc:288] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d73d7d3e291844639d88b7ce89793aaa (127.15.134.194:35583), eaaaa27d00d3443faf6808a5cfe6d7ee (127.15.134.193:37011)
I20260928 08:33:33.154086 29273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:33.148320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:60939 (local address 127.15.134.193:37011)
0928 08:33:33.148539 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:33.148543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:33.148821 (+ 278us) server_negotiation.cc:407] Connection header received
0928 08:33:33.149083 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:33.149089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:33.149164 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:33.149330 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:33:33.149880 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:33.150726 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:33.152370 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:33.153191 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:33.153358 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:33.153468 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:33.153731 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 08:33:33.153854 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 08:33:33.154125 29258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:33.148185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:37011 (local address 127.15.134.195:60939)
0928 08:33:33.148691 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:33.148716 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:33.148887 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:33.149341 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:33.149346 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:33.149377 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:33:33.149701 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:33.149713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:33.150871 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:33.150876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:33.152208 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:33.152220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:33.153354 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:33.153361 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:33.153551 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:33.153576 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 08:33:33.153704 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 08:33:33.153839 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"spinlock_wait_cycles":27008}
I20260928 08:33:33.154435 29271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:33.148195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:40729 (local address 127.15.134.194:35583)
0928 08:33:33.148420 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:33.148427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:33.149373 (+ 946us) server_negotiation.cc:407] Connection header received
0928 08:33:33.149644 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:33.149649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:33.149720 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:33.149823 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:33:33.150383 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:33.151094 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:33.152741 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:33.153473 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:33.153693 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:33.153762 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:33.154038 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 08:33:33.154159 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 08:33:33.154541 29283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:33.148238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:35583 (local address 127.15.134.195:40729)
0928 08:33:33.149286 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:33.149304 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:33.149482 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:33.149838 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:33.149842 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:33.149864 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:33:33.150234 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:33.150245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:33.151245 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:33.151250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:33.152562 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:33.152574 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:33.153712 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:33.153719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:33.153890 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:33.153912 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 08:33:33.154015 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 08:33:33.154144 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":915,"mutex_wait_us":326,"thread_start_us":344,"threads_started":1}
I20260928 08:33:33.155133 28922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955e3b00b02449209eddf80a22c50899" candidate_uuid: "f0dfd0880b7b45beb6f01a39c8e59c33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaaa27d00d3443faf6808a5cfe6d7ee" is_pre_election: true
I20260928 08:33:33.155519 29066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955e3b00b02449209eddf80a22c50899" candidate_uuid: "f0dfd0880b7b45beb6f01a39c8e59c33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d7d3e291844639d88b7ce89793aaa" is_pre_election: true
I20260928 08:33:33.155731 28922 raft_consensus.cc:2454] T 955e3b00b02449209eddf80a22c50899 P eaaaa27d00d3443faf6808a5cfe6d7ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0dfd0880b7b45beb6f01a39c8e59c33 in term 0.
I20260928 08:33:33.156054 29066 raft_consensus.cc:2454] T 955e3b00b02449209eddf80a22c50899 P d73d7d3e291844639d88b7ce89793aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0dfd0880b7b45beb6f01a39c8e59c33 in term 0.
I20260928 08:33:33.156445 29136 leader_election.cc:302] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaaaa27d00d3443faf6808a5cfe6d7ee, f0dfd0880b7b45beb6f01a39c8e59c33; no voters:
I20260928 08:33:33.156841 29281 raft_consensus.cc:2781] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:33:33.156984 29281 raft_consensus.cc:482] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:33:33.157094 29281 raft_consensus.cc:3037] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:33:33.159307 29281 raft_consensus.cc:504] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dfd0880b7b45beb6f01a39c8e59c33" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 42823 } } peers { permanent_uuid: "d73d7d3e291844639d88b7ce89793aaa" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35583 } } peers { permanent_uuid: "eaaaa27d00d3443faf6808a5cfe6d7ee" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 37011 } }
I20260928 08:33:33.159901 29281 leader_election.cc:288] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33 [CANDIDATE]: Term 1 election: Requested vote from peers d73d7d3e291844639d88b7ce89793aaa (127.15.134.194:35583), eaaaa27d00d3443faf6808a5cfe6d7ee (127.15.134.193:37011)
I20260928 08:33:33.160537 29066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955e3b00b02449209eddf80a22c50899" candidate_uuid: "f0dfd0880b7b45beb6f01a39c8e59c33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d73d7d3e291844639d88b7ce89793aaa"
I20260928 08:33:33.160519 28922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955e3b00b02449209eddf80a22c50899" candidate_uuid: "f0dfd0880b7b45beb6f01a39c8e59c33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaaa27d00d3443faf6808a5cfe6d7ee"
I20260928 08:33:33.160777 29066 raft_consensus.cc:3037] T 955e3b00b02449209eddf80a22c50899 P d73d7d3e291844639d88b7ce89793aaa [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:33:33.160776 28922 raft_consensus.cc:3037] T 955e3b00b02449209eddf80a22c50899 P eaaaa27d00d3443faf6808a5cfe6d7ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:33:33.164075 28922 raft_consensus.cc:2454] T 955e3b00b02449209eddf80a22c50899 P eaaaa27d00d3443faf6808a5cfe6d7ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0dfd0880b7b45beb6f01a39c8e59c33 in term 1.
I20260928 08:33:33.164074 29066 raft_consensus.cc:2454] T 955e3b00b02449209eddf80a22c50899 P d73d7d3e291844639d88b7ce89793aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0dfd0880b7b45beb6f01a39c8e59c33 in term 1.
I20260928 08:33:33.164567 29136 leader_election.cc:302] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eaaaa27d00d3443faf6808a5cfe6d7ee, f0dfd0880b7b45beb6f01a39c8e59c33; no voters:
I20260928 08:33:33.165001 29281 raft_consensus.cc:2781] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:33:33.165396 29281 raft_consensus.cc:686] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33 [term 1 LEADER]: Becoming Leader. State: Replica: f0dfd0880b7b45beb6f01a39c8e59c33, State: Running, Role: LEADER
I20260928 08:33:33.165947 29281 consensus_queue.cc:237] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dfd0880b7b45beb6f01a39c8e59c33" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 42823 } } peers { permanent_uuid: "d73d7d3e291844639d88b7ce89793aaa" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35583 } } peers { permanent_uuid: "eaaaa27d00d3443faf6808a5cfe6d7ee" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 37011 } }
I20260928 08:33:33.171455 28769 catalog_manager.cc:5495] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33 reported cstate change: term changed from 0 to 1, leader changed from <none> to f0dfd0880b7b45beb6f01a39c8e59c33 (127.15.134.195). New cstate: current_term: 1 leader_uuid: "f0dfd0880b7b45beb6f01a39c8e59c33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0dfd0880b7b45beb6f01a39c8e59c33" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 42823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d73d7d3e291844639d88b7ce89793aaa" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaaaa27d00d3443faf6808a5cfe6d7ee" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 37011 } health_report { overall_health: UNKNOWN } } }
I20260928 08:33:33.255837 29270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:33.247552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44324 (local address 127.15.134.195:42823)
0928 08:33:33.247755 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:33.247762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:33.247842 (+ 80us) server_negotiation.cc:407] Connection header received
0928 08:33:33.248202 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:33.248210 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:33.248349 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:33.248516 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:33:33.249038 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:33.250013 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:33.252379 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:33.253313 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:33.253470 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:33.254229 (+ 759us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:33:33.254962 (+ 733us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:33:33.255148 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:33.255450 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 08:33:33.255541 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 08:33:33.267554 29066 raft_consensus.cc:1261] T 955e3b00b02449209eddf80a22c50899 P d73d7d3e291844639d88b7ce89793aaa [term 1 FOLLOWER]: Refusing update from remote peer f0dfd0880b7b45beb6f01a39c8e59c33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:33:33.267552 28922 raft_consensus.cc:1261] T 955e3b00b02449209eddf80a22c50899 P eaaaa27d00d3443faf6808a5cfe6d7ee [term 1 FOLLOWER]: Refusing update from remote peer f0dfd0880b7b45beb6f01a39c8e59c33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:33:33.268544 29281 consensus_queue.cc:1035] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaaaa27d00d3443faf6808a5cfe6d7ee" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 37011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:33:33.268944 29284 consensus_queue.cc:1035] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d73d7d3e291844639d88b7ce89793aaa" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:33:33.282713 29288 mvcc.cc:205] Tried to move back new op lower bound from 7334233756731461632 to 7334233756331593728. Current Snapshot: MvccSnapshot[applied={T|T < 7334233756731461632}]
I20260928 08:33:33.305411 28769 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:33:33.305749 28769 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:33:33.309852 28769 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e343dcafe71485180a75c406a699a88: Sending DeleteTablet for 3 replicas of tablet 955e3b00b02449209eddf80a22c50899
I20260928 08:33:33.311057 29181 tablet_service.cc:1514] Processing DeleteTablet for tablet 955e3b00b02449209eddf80a22c50899 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:33:33 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:44308
I20260928 08:33:33.311079 29046 tablet_service.cc:1514] Processing DeleteTablet for tablet 955e3b00b02449209eddf80a22c50899 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:33:33 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:45018
I20260928 08:33:33.311787 28902 tablet_service.cc:1514] Processing DeleteTablet for tablet 955e3b00b02449209eddf80a22c50899 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:33:33 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:51248
I20260928 08:33:33.312812 29300 tablet_replica.cc:330] T 955e3b00b02449209eddf80a22c50899 P eaaaa27d00d3443faf6808a5cfe6d7ee: stopping tablet replica
I20260928 08:33:33.313575 29300 raft_consensus.cc:2229] T 955e3b00b02449209eddf80a22c50899 P eaaaa27d00d3443faf6808a5cfe6d7ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:33:33.314707 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 28806
I20260928 08:33:33.321524 29298 tablet_replica.cc:330] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33: stopping tablet replica
I20260928 08:33:33.323266 29298 raft_consensus.cc:2229] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:33:33.324132 29298 raft_consensus.cc:2258] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:33:33.324414 29300 raft_consensus.cc:2258] T 955e3b00b02449209eddf80a22c50899 P eaaaa27d00d3443faf6808a5cfe6d7ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:33:33.326205 29299 tablet_replica.cc:330] T 955e3b00b02449209eddf80a22c50899 P d73d7d3e291844639d88b7ce89793aaa: stopping tablet replica
I20260928 08:33:33.326728 29299 raft_consensus.cc:2229] T 955e3b00b02449209eddf80a22c50899 P d73d7d3e291844639d88b7ce89793aaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:33:33.327342 29299 raft_consensus.cc:2258] T 955e3b00b02449209eddf80a22c50899 P d73d7d3e291844639d88b7ce89793aaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:33:33.327458 29300 ts_tablet_manager.cc:1905] T 955e3b00b02449209eddf80a22c50899 P eaaaa27d00d3443faf6808a5cfe6d7ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:33:33.329485 29299 ts_tablet_manager.cc:1905] T 955e3b00b02449209eddf80a22c50899 P d73d7d3e291844639d88b7ce89793aaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:33:33.329492 29298 ts_tablet_manager.cc:1905] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:33:33.335044 29300 ts_tablet_manager.cc:1918] T 955e3b00b02449209eddf80a22c50899 P eaaaa27d00d3443faf6808a5cfe6d7ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:33:33.335318 29300 log.cc:1196] T 955e3b00b02449209eddf80a22c50899 P eaaaa27d00d3443faf6808a5cfe6d7ee: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-0/wal/wals/955e3b00b02449209eddf80a22c50899
I20260928 08:33:33.336027 29300 ts_tablet_manager.cc:1939] T 955e3b00b02449209eddf80a22c50899 P eaaaa27d00d3443faf6808a5cfe6d7ee: Deleting consensus metadata
I20260928 08:33:33.337353 29299 ts_tablet_manager.cc:1918] T 955e3b00b02449209eddf80a22c50899 P d73d7d3e291844639d88b7ce89793aaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:33:33.337612 29299 log.cc:1196] T 955e3b00b02449209eddf80a22c50899 P d73d7d3e291844639d88b7ce89793aaa: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/955e3b00b02449209eddf80a22c50899
I20260928 08:33:33.338141 28755 catalog_manager.cc:4841] TS eaaaa27d00d3443faf6808a5cfe6d7ee (127.15.134.193:37011): tablet 955e3b00b02449209eddf80a22c50899 (table test-table [id=1c18d5ac998145dfb38ce1350461d528]) successfully deleted
I20260928 08:33:33.338241 29299 ts_tablet_manager.cc:1939] T 955e3b00b02449209eddf80a22c50899 P d73d7d3e291844639d88b7ce89793aaa: Deleting consensus metadata
I20260928 08:33:33.340046 28754 catalog_manager.cc:4841] TS d73d7d3e291844639d88b7ce89793aaa (127.15.134.194:35583): tablet 955e3b00b02449209eddf80a22c50899 (table test-table [id=1c18d5ac998145dfb38ce1350461d528]) successfully deleted
I20260928 08:33:33.342310 29298 ts_tablet_manager.cc:1918] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:33:33.342512 29298 log.cc:1196] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/955e3b00b02449209eddf80a22c50899
I20260928 08:33:33.343163 29298 ts_tablet_manager.cc:1939] T 955e3b00b02449209eddf80a22c50899 P f0dfd0880b7b45beb6f01a39c8e59c33: Deleting consensus metadata
I20260928 08:33:33.345120 28753 catalog_manager.cc:4841] TS f0dfd0880b7b45beb6f01a39c8e59c33 (127.15.134.195:42823): tablet 955e3b00b02449209eddf80a22c50899 (table test-table [id=1c18d5ac998145dfb38ce1350461d528]) successfully deleted
I20260928 08:33:33.352178 29273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:33.318181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51262 (local address 127.15.134.193:37011)
0928 08:33:33.318418 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:33.318442 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:33.330617 (+ 12175us) server_negotiation.cc:407] Connection header received
0928 08:33:33.336505 (+ 5888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:33.336512 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:33.336603 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:33.336696 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:33.338812 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:33.339871 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:33.341771 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:33.342392 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:33.342636 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:33.346149 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:33.346171 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:33.346186 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:33.346222 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:33.349272 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:33.350003 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:33.350011 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:33.350013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:33.350142 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:33.350543 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:33.350548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:33.350551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:33.350937 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:33.351265 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:33.351558 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 08:33:33.351791 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 08:33:33.353008 28962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:33:33.446738 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 28806
I20260928 08:33:33.465408 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 28971
I20260928 08:33:33.485363 29271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:33.466296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45020 (local address 127.15.134.194:35583)
0928 08:33:33.466506 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:33.466512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:33.466535 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:33:33.466709 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:33.466715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:33.466821 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:33.466939 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:33.468243 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:33.469496 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:33.470811 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:33.471471 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:33.471603 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:33.478039 (+ 6436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:33.478061 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:33.478083 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:33.478132 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:33.482206 (+ 4074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:33.483017 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:33.483027 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:33.483031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:33.483209 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:33.483591 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:33.483596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:33.483600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:33.484114 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:33.484428 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:33.484719 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 08:33:33.484957 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 08:33:33.486146 29107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:33:33.585873 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 28971
I20260928 08:33:33.604041 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 29115
I20260928 08:33:33.619149 29270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:33.605009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44340 (local address 127.15.134.195:42823)
0928 08:33:33.605228 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:33.605236 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:33.605260 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:33:33.605555 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:33.605562 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:33.605655 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:33.605789 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:33.607245 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:33.608071 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:33.609575 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:33.610062 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:33.610222 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:33.612914 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:33.612931 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:33.612948 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:33.612993 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:33.616131 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:33.616894 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:33.616914 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:33.616916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:33.617110 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:33.617537 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:33.617541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:33.617543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:33.617910 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:33.618244 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:33.618548 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 08:33:33.618781 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 08:33:33.620016 29248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:33:33.745605 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 29115
I20260928 08:33:33.766371 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 28736
I20260928 08:33:33.767750 28799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:33:33.884796 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 28736
I20260928 08:33:33.907907 15899 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 08:33:34.253407 15899 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 08:33:34.253654 15899 mini_ranger.cc:66] Stopping Ranger...
I20260928 08:33:34.599781 15899 mini_ranger.cc:68] Stopped Ranger
2026-09-28 08:33:34.600 UTC [27787] LOG: received smart shutdown request
2026-09-28 08:33:34.603 UTC [27787] LOG: background worker "logical replication launcher" (PID 27796) exited with exit code 1
2026-09-28 08:33:34.603 UTC [27791] LOG: shutting down
2026-09-28 08:33:34.783 UTC [27787] LOG: database system is shut down
2026-09-28T08:33:34Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54957 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:33:35 dist-test-slave-qp2n krb5kdc[29314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:33:35 dist-test-slave-qp2n krb5kdc[29314](info): set up 2 sockets
Sep 28 08:33:35 dist-test-slave-qp2n krb5kdc[29314](info): commencing operation
krb5kdc: starting...
W20260928 08:33:36.034541 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:33:36 dist-test-slave-qp2n krb5kdc[29314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:33:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:33:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:33:36.285356 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:41729
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:36531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:41729,127.15.134.253:33329,127.15.134.252:42687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:33:36.637557 29327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:33:36.637861 29327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:33:36.638059 29327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:33:36.645623 29327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:33:36.645753 29327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:33:36.645829 29327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:33:36.645928 29327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:33:36.646003 29327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:33:36.656517 29327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:36531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:41729,127.15.134.253:33329,127.15.134.252:42687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:41729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:33:36.658068 29327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:33:36.659492 29327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:33:36.671864 29333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:36.671928 29335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:36.671953 29332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:36.673287 29327 server_base.cc:1034] running on GCE node
I20260928 08:33:36.674180 29327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:33:36.675738 29327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:33:36.677006 29327 hybrid_clock.cc:648] HybridClock initialized: now 1790584416676948 us; error 51 us; skew 500 ppm
I20260928 08:33:36.677420 29327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:33:36 dist-test-slave-qp2n krb5kdc[29314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584416, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:33:36.680783 29327 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:33:36.682456 29327 webserver.cc:460] Webserver started at http://127.15.134.254:41533/ using document root <none> and password file <none>
I20260928 08:33:36.683233 29327 fs_manager.cc:359] Metadata directory not provided
I20260928 08:33:36.683390 29327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:33:36.683723 29327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:33:36.687017 29327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "5cabf1cf7b2e442f8cc6b5827128f046"
format_stamp: "Formatted at 2026-09-28 08:33:36 on dist-test-slave-qp2n"
server_key: "615cda03bf3ffd3873bd4ce5c54d5100"
server_key_iv: "069e43e5e12bdc93fce1b896311a6c77"
server_key_version: "encryptionkey@0"
I20260928 08:33:36.687927 29327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "5cabf1cf7b2e442f8cc6b5827128f046"
format_stamp: "Formatted at 2026-09-28 08:33:36 on dist-test-slave-qp2n"
server_key: "615cda03bf3ffd3873bd4ce5c54d5100"
server_key_iv: "069e43e5e12bdc93fce1b896311a6c77"
server_key_version: "encryptionkey@0"
I20260928 08:33:36.693436 29327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 08:33:36.697103 29342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:36.698479 29327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 08:33:36.698669 29327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "5cabf1cf7b2e442f8cc6b5827128f046"
format_stamp: "Formatted at 2026-09-28 08:33:36 on dist-test-slave-qp2n"
server_key: "615cda03bf3ffd3873bd4ce5c54d5100"
server_key_iv: "069e43e5e12bdc93fce1b896311a6c77"
server_key_version: "encryptionkey@0"
I20260928 08:33:36.698874 29327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:33:36.719429 29327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:33:36.723899 29327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:33:36.724207 29327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:33:36.744422 29327 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:41729
I20260928 08:33:36.744472 29393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:41729 every 8 connection(s)
I20260928 08:33:36.745911 29327 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:33:36.749125 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 29327
I20260928 08:33:36.749354 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:33:36.749979 29394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:33:36.750052 15899 external_mini_cluster.cc:1428] Setting key 4b76f0299515d712599766cfef677b2a
I20260928 08:33:36.756973 29394 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41729 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.134.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.253@KRBTEST.COM" created.
Sep 28 08:33:36 dist-test-slave-qp2n krb5kdc[29314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584416, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:33:36.782227 29397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:36.760391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54756 (local address 127.15.134.254:41729)
0928 08:33:36.761371 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:36.761389 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:36.761893 (+ 504us) server_negotiation.cc:407] Connection header received
0928 08:33:36.763968 (+ 2075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:36.763974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:36.764206 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:36.764432 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:36.765857 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:36.767133 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:36.768603 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:36.769200 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:36.769408 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:36.774544 (+ 5136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:36.774553 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:36.774567 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:36.774603 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:36.777577 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:36.778323 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:36.778330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:36.778332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:36.778492 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:36.778900 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:36.778906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:36.778911 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:36.779448 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:36.779665 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:36.780241 (+ 576us) server_negotiation.cc:299] Negotiation successful
0928 08:33:36.780671 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":734,"thread_start_us":518,"threads_started":1}
I20260928 08:33:36.782248 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:36.760399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41729 (local address 127.0.0.1:54756)
0928 08:33:36.761721 (+ 1322us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:36.761758 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:36.763567 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:36.764456 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:36.764462 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:36.765301 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:33:36.765644 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:36.765662 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:36.767300 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:36.767306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:36.768443 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:36.768456 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:36.769448 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:36.769455 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:36.769631 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:36.770399 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:33:36.770421 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:33:36.774260 (+ 3839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:33:36.777737 (+ 3477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:36.777749 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:36.777763 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:36.778171 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:36.778607 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:36.778613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:36.778617 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:36.778780 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:36.779598 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:33:36.779606 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:33:36.779841 (+ 235us) client_negotiation.cc:769] Sending connection context
0928 08:33:36.780102 (+ 261us) client_negotiation.cc:240] Negotiation successful
0928 08:33:36.780772 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1116,"mutex_wait_us":193,"spinlock_wait_cycles":129280,"thread_start_us":899,"threads_started":1}
I20260928 08:33:36.785413 29394 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:33:36.786501 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:36.786099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:33329 (local address 127.0.0.1:40160)
0928 08:33:36.786267 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:36.786378 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 08:33:36.786751 29344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.134.253:33329: connect: Connection refused (error 111)
I20260928 08:33:36.788381 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:36.788018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:33329 (local address 127.0.0.1:40162)
0928 08:33:36.788173 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:36.788249 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 08:33:36.788658 29394 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.253:33329: Network error: Client connection negotiation failed: client connection to 127.15.134.253:33329: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.253.keytab.
Entry for principal kudu/127.15.134.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.253.keytab.
I20260928 08:33:36.805248 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.253:33329
--webserver_interface=127.15.134.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.253.keytab
--principal=kudu/127.15.134.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:36531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:41729,127.15.134.253:33329,127.15.134.252:42687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:33:36.838027 29394 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } attempt: 1
I20260928 08:33:36.839437 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:36.838816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:33329 (local address 127.0.0.1:40174)
0928 08:33:36.839068 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:36.839258 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 08:33:36.840597 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:36.840175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:33329 (local address 127.0.0.1:40190)
0928 08:33:36.840337 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:36.840429 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 08:33:36.840893 29394 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.253:33329: Network error: Client connection negotiation failed: client connection to 127.15.134.253:33329: connect: Connection refused (error 111)
I20260928 08:33:36.909168 29394 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } attempt: 2
I20260928 08:33:36.910467 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:36.909874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:33329 (local address 127.0.0.1:40192)
0928 08:33:36.910161 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:36.910293 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 08:33:36.911564 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:36.911147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:33329 (local address 127.0.0.1:40196)
0928 08:33:36.911311 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:36.911398 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 08:33:36.911777 29394 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.253:33329: Network error: Client connection negotiation failed: client connection to 127.15.134.253:33329: connect: Connection refused (error 111)
I20260928 08:33:37.003022 29394 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } attempt: 3
I20260928 08:33:37.004324 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.003746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:33329 (local address 127.0.0.1:40204)
0928 08:33:37.003996 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.004123 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 08:33:37.005571 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.005052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:33329 (local address 127.0.0.1:40208)
0928 08:33:37.005245 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.005372 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 08:33:37.005776 29394 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.253:33329: Network error: Client connection negotiation failed: client connection to 127.15.134.253:33329: connect: Connection refused (error 111)
I20260928 08:33:37.149011 29394 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } attempt: 4
I20260928 08:33:37.150319 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.149677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:33329 (local address 127.0.0.1:40212)
0928 08:33:37.150009 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.150131 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260928 08:33:37.151428 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.151015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:33329 (local address 127.0.0.1:40222)
0928 08:33:37.151176 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.151280 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:33329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 08:33:37.151620 29394 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.253:33329: Network error: Client connection negotiation failed: client connection to 127.15.134.253:33329: connect: Connection refused (error 111)
W20260928 08:33:37.155010 29401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:33:37.155314 29401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:33:37.155488 29401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:33:37.163055 29401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:33:37.163188 29401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:33:37.163300 29401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:33:37.163398 29401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:33:37.163476 29401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:33:37.173810 29401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:36531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:41729,127.15.134.253:33329,127.15.134.252:42687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.253.keytab
--principal=kudu/127.15.134.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.253:33329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.134.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:33:37.175232 29401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:33:37.176623 29401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:33:37.191124 29407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:37.191195 29409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:37.191124 29406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:37.192013 29401 server_base.cc:1034] running on GCE node
I20260928 08:33:37.192884 29401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:33:37.194438 29401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:33:37.195717 29401 hybrid_clock.cc:648] HybridClock initialized: now 1790584417195637 us; error 66 us; skew 500 ppm
I20260928 08:33:37.196238 29401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:33:37 dist-test-slave-qp2n krb5kdc[29314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584417, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:33:37.199636 29401 init.cc:377] Logged in from keytab as kudu/127.15.134.253@KRBTEST.COM (short username kudu)
I20260928 08:33:37.201239 29401 webserver.cc:460] Webserver started at http://127.15.134.253:34739/ using document root <none> and password file <none>
I20260928 08:33:37.201985 29401 fs_manager.cc:359] Metadata directory not provided
I20260928 08:33:37.202116 29401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:33:37.202490 29401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:33:37.205628 29401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-1/data/instance:
uuid: "fd9937e8256d45ae9dff11b9b078eaeb"
format_stamp: "Formatted at 2026-09-28 08:33:37 on dist-test-slave-qp2n"
server_key: "10a6c148642b86fbea013c0ff38c0d62"
server_key_iv: "ffa3f2a8c0e7674545ef5bca9562066d"
server_key_version: "encryptionkey@0"
I20260928 08:33:37.206462 29401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-1/wal/instance:
uuid: "fd9937e8256d45ae9dff11b9b078eaeb"
format_stamp: "Formatted at 2026-09-28 08:33:37 on dist-test-slave-qp2n"
server_key: "10a6c148642b86fbea013c0ff38c0d62"
server_key_iv: "ffa3f2a8c0e7674545ef5bca9562066d"
server_key_version: "encryptionkey@0"
I20260928 08:33:37.211930 29401 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:33:37.215636 29416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:37.217308 29401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:33:37.217499 29401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-1/wal
uuid: "fd9937e8256d45ae9dff11b9b078eaeb"
format_stamp: "Formatted at 2026-09-28 08:33:37 on dist-test-slave-qp2n"
server_key: "10a6c148642b86fbea013c0ff38c0d62"
server_key_iv: "ffa3f2a8c0e7674545ef5bca9562066d"
server_key_version: "encryptionkey@0"
I20260928 08:33:37.217801 29401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:33:37.229135 29401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:33:37.233229 29401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:33:37.233613 29401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:33:37.252238 29401 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.253:33329
I20260928 08:33:37.252261 29467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.253:33329 every 8 connection(s)
I20260928 08:33:37.253824 29401 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-1/data/info.pb
I20260928 08:33:37.257643 29468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:33:37.258850 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 29401
I20260928 08:33:37.259056 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-1/wal/instance
I20260928 08:33:37.259547 15899 external_mini_cluster.cc:1428] Setting key 3a8ceb624e01acd1c02b1625d9a62748
I20260928 08:33:37.264492 29468 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41729 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.134.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.252@KRBTEST.COM" created.
Sep 28 08:33:37 dist-test-slave-qp2n krb5kdc[29314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584417, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.253@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:33:37.290045 29397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.267487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54770 (local address 127.15.134.254:41729)
0928 08:33:37.267706 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:37.267713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:37.268973 (+ 1260us) server_negotiation.cc:407] Connection header received
0928 08:33:37.270830 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:37.270839 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:37.270946 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:37.271075 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:37.273015 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.274177 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.276062 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.276905 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.277071 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:37.283739 (+ 6668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:37.283746 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:37.283749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:37.283784 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:37.286579 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:37.287590 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:37.287594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:37.287596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:37.287716 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:37.288199 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:37.288203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:37.288205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:37.288479 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:37.288588 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:37.289483 (+ 895us) server_negotiation.cc:299] Negotiation successful
0928 08:33:37.289720 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 08:33:37.291062 29471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.267952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41729 (local address 127.0.0.1:54770)
0928 08:33:37.268800 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.268846 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:37.270554 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:37.271421 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:37.271430 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:37.272093 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:33:37.272806 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:37.272828 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.274362 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.274366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:37.275894 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:37.275908 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.277145 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.277150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:37.277368 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:37.278168 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:33:37.278192 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:33:37.283494 (+ 5302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:33:37.286747 (+ 3253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:37.286759 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:37.286780 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:37.287453 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:37.287832 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:37.287840 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:37.287846 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:37.288085 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:37.288635 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:33:37.288648 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:33:37.289063 (+ 415us) client_negotiation.cc:769] Sending connection context
0928 08:33:37.289339 (+ 276us) client_negotiation.cc:240] Negotiation successful
0928 08:33:37.289876 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":362,"threads_started":1}
I20260928 08:33:37.293150 29468 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:33:37 dist-test-slave-qp2n krb5kdc[29314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584417, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.253@KRBTEST.COM for kudu/127.15.134.253@KRBTEST.COM
I20260928 08:33:37.309877 29471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.293936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:33329 (local address 127.0.0.1:40238)
0928 08:33:37.294172 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.294193 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:37.294299 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:37.295271 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:37.295279 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:37.295818 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:33:37.296045 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:37.296059 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.297519 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.297527 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:37.299158 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:37.299177 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.300211 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.300220 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:37.300473 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:37.301202 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:33:37.301241 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:33:37.303892 (+ 2651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:33:37.307111 (+ 3219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:37.307117 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:37.307121 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:37.307477 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:37.307948 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:37.307955 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:37.307958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:37.308094 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:37.308855 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:33:37.308862 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:33:37.309018 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 08:33:37.309189 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 08:33:37.309488 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 08:33:37.309984 29473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.294177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40238 (local address 127.15.134.253:33329)
0928 08:33:37.294846 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:37.294852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:37.294877 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:33:37.294919 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:37.294924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:37.295102 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:37.295213 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:37.296228 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.297356 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.299313 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.300045 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.300145 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:37.304101 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:37.304113 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:37.304126 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:37.304159 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:37.306971 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:37.307616 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:37.307628 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:37.307632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:37.307841 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:37.308209 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:37.308214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:37.308219 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:37.308736 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:37.308947 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:37.309338 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 08:33:37.309600 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":503,"thread_start_us":369,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.252.keytab.
Entry for principal kudu/127.15.134.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.252.keytab.
I20260928 08:33:37.312358 29468 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:33:37.313287 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.252:42687
--webserver_interface=127.15.134.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.252.keytab
--principal=kudu/127.15.134.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:36531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:41729,127.15.134.253:33329,127.15.134.252:42687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:33:37.313733 29471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.313170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:42687 (local address 127.0.0.1:36314)
0928 08:33:37.313371 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.313522 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260928 08:33:37.314059 29419 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
I20260928 08:33:37.315773 29471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.315304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:42687 (local address 127.0.0.1:36328)
0928 08:33:37.315512 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.315614 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 08:33:37.316038 29468 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:42687: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
I20260928 08:33:37.365402 29468 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } attempt: 1
I20260928 08:33:37.366838 29471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.366241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:42687 (local address 127.0.0.1:36336)
0928 08:33:37.366467 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.366585 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 08:33:37.368094 29471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.367589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:42687 (local address 127.0.0.1:36342)
0928 08:33:37.367882 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.367967 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 08:33:37.368340 29468 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:42687: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
I20260928 08:33:37.436602 29468 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } attempt: 2
I20260928 08:33:37.437831 29471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.437353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:42687 (local address 127.0.0.1:36358)
0928 08:33:37.437557 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.437657 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 08:33:37.438841 29471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.438492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:42687 (local address 127.0.0.1:36370)
0928 08:33:37.438641 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.438720 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 08:33:37.439055 29468 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:42687: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
I20260928 08:33:37.450899 29394 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } attempt: 5
Sep 28 08:33:37 dist-test-slave-qp2n krb5kdc[29314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584416, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for kudu/127.15.134.253@KRBTEST.COM
I20260928 08:33:37.466956 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.451663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:33329 (local address 127.0.0.1:47392)
0928 08:33:37.451864 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.451887 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:37.452004 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:37.452415 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:37.452420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:37.452946 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:33:37.453155 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:37.453166 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.454722 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.454726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:37.455816 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:37.455832 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.456896 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.456901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:37.457048 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:37.457826 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:33:37.457849 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:33:37.460348 (+ 2499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:33:37.464230 (+ 3882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:37.464234 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:37.464237 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:37.464731 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:37.465146 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:37.465149 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:37.465151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:37.465335 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:37.466024 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:33:37.466028 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:33:37.466173 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 08:33:37.466333 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 08:33:37.466635 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 08:33:37.467052 29473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.451790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47392 (local address 127.15.134.253:33329)
0928 08:33:37.451991 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:37.451999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:37.452025 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:33:37.452111 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:37.452117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:37.452196 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:37.452320 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:37.453397 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.454547 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.455987 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.456691 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.456823 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:37.460595 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:37.460604 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:37.460648 (+ 44us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:37.460703 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:37.464042 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:37.464864 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:37.464870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:37.464874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:37.465031 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:37.465450 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:37.465455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:37.465458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:37.465829 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:37.465987 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:37.466479 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 08:33:37.466716 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 08:33:37.468425 29394 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:33:37.469836 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.469369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:42687 (local address 127.0.0.1:36384)
0928 08:33:37.469554 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.469635 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 08:33:37.471171 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.470645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:42687 (local address 127.0.0.1:36386)
0928 08:33:37.470877 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.471016 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260928 08:33:37.471460 29394 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:42687: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
I20260928 08:33:37.522791 29394 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } attempt: 1
I20260928 08:33:37.523917 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.523446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:42687 (local address 127.0.0.1:36392)
0928 08:33:37.523656 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.523753 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 08:33:37.524896 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.524549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:42687 (local address 127.0.0.1:36402)
0928 08:33:37.524730 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.524801 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 08:33:37.525117 29394 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:42687: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
I20260928 08:33:37.530304 29468 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } attempt: 3
I20260928 08:33:37.531445 29471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.530966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:42687 (local address 127.0.0.1:36416)
0928 08:33:37.531195 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.531281 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 08:33:37.532790 29471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.532261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:42687 (local address 127.0.0.1:36432)
0928 08:33:37.532510 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.532595 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260928 08:33:37.533102 29468 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:42687: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
I20260928 08:33:37.593365 29394 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } attempt: 2
I20260928 08:33:37.594607 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.594087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:42687 (local address 127.0.0.1:36442)
0928 08:33:37.594331 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.594427 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 08:33:37.595887 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.595490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:42687 (local address 127.0.0.1:36454)
0928 08:33:37.595622 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.595704 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 08:33:37.596117 29394 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:42687: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
W20260928 08:33:37.657823 29475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:33:37.658116 29475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:33:37.658285 29475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:33:37.665544 29475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:33:37.665638 29475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:33:37.665743 29475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:33:37.665839 29475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:33:37.665922 29475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:33:37.675705 29475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:36531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:41729,127.15.134.253:33329,127.15.134.252:42687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.252.keytab
--principal=kudu/127.15.134.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.252:42687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.134.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:33:37.676353 29468 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } attempt: 4
I20260928 08:33:37.677212 29475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:33:37.677575 29471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.677045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:42687 (local address 127.0.0.1:36468)
0928 08:33:37.677303 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.677427 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 08:33:37.678540 29471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.678215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:42687 (local address 127.0.0.1:36476)
0928 08:33:37.678360 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.678434 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 08:33:37.678574 29475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:33:37.678751 29468 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:42687: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
W20260928 08:33:37.690651 29480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:37.690670 29481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:37.691039 29483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:37.691521 29475 server_base.cc:1034] running on GCE node
I20260928 08:33:37.692376 29475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:33:37.693948 29475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:33:37.695206 29475 hybrid_clock.cc:648] HybridClock initialized: now 1790584417695131 us; error 64 us; skew 500 ppm
I20260928 08:33:37.695571 29475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:33:37 dist-test-slave-qp2n krb5kdc[29314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584417, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:33:37.698949 29475 init.cc:377] Logged in from keytab as kudu/127.15.134.252@KRBTEST.COM (short username kudu)
I20260928 08:33:37.700551 29475 webserver.cc:460] Webserver started at http://127.15.134.252:35657/ using document root <none> and password file <none>
I20260928 08:33:37.701534 29475 fs_manager.cc:359] Metadata directory not provided
I20260928 08:33:37.701676 29475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:33:37.701952 29475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:33:37.702368 29394 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } attempt: 3
I20260928 08:33:37.703603 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.702993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:42687 (local address 127.0.0.1:36480)
0928 08:33:37.703294 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.703392 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260928 08:33:37.704597 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.704267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:42687 (local address 127.0.0.1:36490)
0928 08:33:37.704422 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.704497 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260928 08:33:37.704908 29394 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:42687: Network error: Client connection negotiation failed: client connection to 127.15.134.252:42687: connect: Connection refused (error 111)
I20260928 08:33:37.705866 29475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-2/data/instance:
uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3"
format_stamp: "Formatted at 2026-09-28 08:33:37 on dist-test-slave-qp2n"
server_key: "43b1d833cd693c3bb9f3abf6f68b0a2d"
server_key_iv: "0a3b3c657ec7ed13a558854326a433e7"
server_key_version: "encryptionkey@0"
I20260928 08:33:37.706667 29475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-2/wal/instance:
uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3"
format_stamp: "Formatted at 2026-09-28 08:33:37 on dist-test-slave-qp2n"
server_key: "43b1d833cd693c3bb9f3abf6f68b0a2d"
server_key_iv: "0a3b3c657ec7ed13a558854326a433e7"
server_key_version: "encryptionkey@0"
I20260928 08:33:37.712037 29475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 08:33:37.715946 29490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:37.717643 29475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:33:37.717844 29475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-2/wal
uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3"
format_stamp: "Formatted at 2026-09-28 08:33:37 on dist-test-slave-qp2n"
server_key: "43b1d833cd693c3bb9f3abf6f68b0a2d"
server_key_iv: "0a3b3c657ec7ed13a558854326a433e7"
server_key_version: "encryptionkey@0"
I20260928 08:33:37.718077 29475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:33:37.734828 29475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:33:37.739580 29475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:33:37.739938 29475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:33:37.760231 29475 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.252:42687
I20260928 08:33:37.760254 29541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.252:42687 every 8 connection(s)
I20260928 08:33:37.761783 29475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-2/data/info.pb
I20260928 08:33:37.765722 29542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:33:37.766844 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 29475
I20260928 08:33:37.767046 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790584205252592-15899-0/minicluster-data/master-2/wal/instance
I20260928 08:33:37.767556 15899 external_mini_cluster.cc:1428] Setting key 699bf219e743161193d981dcdca12007
I20260928 08:33:37.775836 29542 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41729 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:33:37 dist-test-slave-qp2n krb5kdc[29314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.252@KRBTEST.COM
Sep 28 08:33:37 dist-test-slave-qp2n krb5kdc[29314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584417, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.252@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:33:37.794414 29545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.769652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36506 (local address 127.15.134.252:42687)
0928 08:33:37.770544 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:37.770559 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:37.770611 (+ 52us) server_negotiation.cc:407] Connection header received
0928 08:33:37.772544 (+ 1933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:37.772575 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:37.773159 (+ 584us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:37.773650 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:37.775057 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.777114 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.779130 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.780383 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.780580 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:37.784956 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:37.784965 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:37.784977 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:37.785018 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:37.788794 (+ 3776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:37.789625 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:37.789637 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:37.789650 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:37.790035 (+ 385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:37.790487 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:37.790496 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:37.790502 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:37.791397 (+ 895us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:37.791756 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:37.792157 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 08:33:37.792762 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":378,"threads_started":1}
I20260928 08:33:37.795662 29397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.778951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56034 (local address 127.15.134.254:41729)
0928 08:33:37.779161 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:37.779167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:37.780109 (+ 942us) server_negotiation.cc:407] Connection header received
0928 08:33:37.780351 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:37.780355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:37.780432 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:37.780564 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:37.781742 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.782438 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.783782 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.784220 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.784348 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:37.788589 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:37.788599 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:37.788635 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:37.788685 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:37.792345 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:37.793220 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:37.793227 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:37.793231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:37.793421 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:37.793898 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:37.793903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:37.793906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:37.794283 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:37.794484 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:37.795053 (+ 569us) server_negotiation.cc:299] Negotiation successful
0928 08:33:37.795277 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 08:33:37.795707 29546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.779054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41729 (local address 127.0.0.1:56034)
0928 08:33:37.779919 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.779970 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:37.780138 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:37.780644 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:37.780652 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:37.781322 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:33:37.781577 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:37.781586 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.782590 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.782595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:37.783590 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:37.783601 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.784453 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.784461 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:37.784682 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:37.785377 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:33:37.785394 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:33:37.788313 (+ 2919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:33:37.792496 (+ 4183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:37.792506 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:37.792511 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:37.793031 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:37.793544 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:37.793547 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:37.793549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:37.793672 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:37.794429 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:33:37.794466 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:33:37.794684 (+ 218us) client_negotiation.cc:769] Sending connection context
0928 08:33:37.794934 (+ 250us) client_negotiation.cc:240] Negotiation successful
0928 08:33:37.795284 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":405,"threads_started":1}
I20260928 08:33:37.797186 29542 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:33:37 dist-test-slave-qp2n krb5kdc[29314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584417, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.252@KRBTEST.COM for kudu/127.15.134.253@KRBTEST.COM
Sep 28 08:33:37 dist-test-slave-qp2n krb5kdc[29314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:33:37.812074 29546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.797840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:33329 (local address 127.0.0.1:47404)
0928 08:33:37.798004 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.798024 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:37.798119 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:37.798606 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:37.798610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:37.798988 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:33:37.799136 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:37.799144 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.800445 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.800450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:37.801660 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:37.801672 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.802808 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.802814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:37.802991 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:37.803586 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:33:37.803609 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:33:37.805787 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:33:37.809681 (+ 3894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:37.809685 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:37.809687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:37.810198 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:37.810621 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:37.810624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:37.810626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:37.810756 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:37.811409 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:33:37.811412 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:33:37.811543 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 08:33:37.811654 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 08:33:37.811838 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 08:33:37.812408 29473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.798087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47404 (local address 127.15.134.253:33329)
0928 08:33:37.798286 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:37.798291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:37.798308 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:33:37.798369 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:37.798386 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:37.798469 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:37.798541 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:37.799296 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.800310 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.801855 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.802628 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.802782 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:37.806003 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:37.806012 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:37.806018 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:37.806068 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:37.809549 (+ 3481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:37.810325 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:37.810333 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:37.810338 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:37.810502 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:37.810868 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:37.810876 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:37.810881 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:37.811263 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:37.811421 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:37.811794 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 08:33:37.812008 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 08:33:37.813493 29542 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:33:37.815816 29397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.798094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56036 (local address 127.15.134.254:41729)
0928 08:33:37.798293 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:37.798297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:37.798310 (+ 13us) server_negotiation.cc:407] Connection header received
0928 08:33:37.798436 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:37.798439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:37.798490 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:37.798603 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:37.800108 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.801225 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.803156 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.803920 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.804047 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:37.809118 (+ 5071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:37.809127 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:37.809133 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:37.809188 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:37.812761 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:37.813530 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:37.813534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:37.813536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:37.813696 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:37.814287 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:37.814291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:37.814293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:37.814662 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:37.814920 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:37.815269 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 08:33:37.815490 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
Sep 28 08:33:37 dist-test-slave-qp2n krb5kdc[29314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584417, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.252@KRBTEST.COM for kudu/127.15.134.252@KRBTEST.COM
Sep 28 08:33:37 dist-test-slave-qp2n krb5kdc[29314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584416, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.253@KRBTEST.COM
I20260928 08:33:37.828473 29546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.814390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:42687 (local address 127.0.0.1:36514)
0928 08:33:37.814574 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.814593 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:37.814684 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:37.815051 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:37.815054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:37.815521 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:33:37.815688 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:37.815699 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.816951 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.816956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:37.818180 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:37.818194 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.819263 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.819267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:37.819420 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:37.819967 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:33:37.819985 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:33:37.822469 (+ 2484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:33:37.825672 (+ 3203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:37.825680 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:37.825686 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:37.826279 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:37.826665 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:37.826671 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:37.826676 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:37.826867 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:37.827513 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:33:37.827516 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:33:37.827643 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 08:33:37.827933 (+ 290us) client_negotiation.cc:240] Negotiation successful
0928 08:33:37.828159 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":1}
I20260928 08:33:37.828794 29545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.814571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36514 (local address 127.15.134.252:42687)
0928 08:33:37.814755 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:37.814761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:37.814777 (+ 16us) server_negotiation.cc:407] Connection header received
0928 08:33:37.814825 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:37.814830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:37.814906 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:37.815017 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:37.815853 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.816809 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.818325 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.819124 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.819321 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:37.822721 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:37.822728 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:37.822733 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:37.822771 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:37.825533 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:37.826426 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:37.826430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:37.826432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:37.826539 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:37.826990 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:37.826993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:37.826995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:37.827363 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:37.827536 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:37.828063 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 08:33:37.828318 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":70}
I20260928 08:33:37.834335 29542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3: Bootstrap starting.
I20260928 08:33:37.834379 29473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.817515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47416 (local address 127.15.134.253:33329)
0928 08:33:37.817678 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:37.817684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:37.817734 (+ 50us) server_negotiation.cc:407] Connection header received
0928 08:33:37.817904 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:37.817909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:37.817979 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:37.818077 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:37.819787 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.820515 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.822525 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.823156 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.823347 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:37.828835 (+ 5488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:37.828842 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:37.828846 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:37.828898 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:37.831738 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:37.832400 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:37.832408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:37.832413 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:37.832555 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:37.832964 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:37.832970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:37.832973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:37.833239 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:37.833521 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:37.833794 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 08:33:37.834036 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:33:37.838404 29542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3: Neither blocks nor log segments found. Creating new log.
I20260928 08:33:37.839700 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:33:37.839938 29542 log.cc:824] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3: Log is configured to *not* fsync() on all Append() calls
I20260928 08:33:37.843083 29542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3: No bootstrap required, opened a new log
I20260928 08:33:37.857378 29542 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cabf1cf7b2e442f8cc6b5827128f046" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41729 } } peers { permanent_uuid: "fd9937e8256d45ae9dff11b9b078eaeb" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } } peers { permanent_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } }
I20260928 08:33:37.858137 29542 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:33:37.858373 29542 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd58b2e0defb45bfa5050c52d9fbf8d3, State: Initialized, Role: FOLLOWER
I20260928 08:33:37.859334 29542 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cabf1cf7b2e442f8cc6b5827128f046" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41729 } } peers { permanent_uuid: "fd9937e8256d45ae9dff11b9b078eaeb" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } } peers { permanent_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } }
I20260928 08:33:37.865772 29545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.845257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36530 (local address 127.15.134.252:42687)
0928 08:33:37.845545 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:37.845555 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:37.845585 (+ 30us) server_negotiation.cc:407] Connection header received
0928 08:33:37.845724 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:37.845730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:37.845813 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:37.845946 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:37.847551 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.848392 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.851081 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.852114 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.852257 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:37.857544 (+ 5287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:37.857555 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:37.857563 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:37.857621 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:37.861802 (+ 4181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:37.863008 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:37.863014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:37.863018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:37.863201 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:37.863748 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:37.863753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:37.863757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:37.864189 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:37.864463 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:37.865104 (+ 641us) server_negotiation.cc:299] Negotiation successful
0928 08:33:37.865358 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 08:33:37.867377 29555 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cabf1cf7b2e442f8cc6b5827128f046" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41729 } } peers { permanent_uuid: "fd9937e8256d45ae9dff11b9b078eaeb" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } } peers { permanent_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } } }
I20260928 08:33:37.869231 29397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.844732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56046 (local address 127.15.134.254:41729)
0928 08:33:37.844944 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:37.844954 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:37.849129 (+ 4175us) server_negotiation.cc:407] Connection header received
0928 08:33:37.849416 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:37.849422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:37.849514 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:37.849587 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:37.852352 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.853620 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.856169 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.856904 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.857014 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:37.861534 (+ 4520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:37.861543 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:37.861550 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:37.861599 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:37.865115 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:37.866060 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:37.866066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:37.866069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:37.866243 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:37.866673 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:37.866678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:37.866682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:37.867074 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:37.867309 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:37.868530 (+ 1221us) server_negotiation.cc:299] Negotiation successful
0928 08:33:37.868837 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 08:33:37.870250 29542 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:33:37.870401 29555 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:33:37.870337 29556 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:33:37.871038 29556 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cabf1cf7b2e442f8cc6b5827128f046" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41729 } } peers { permanent_uuid: "fd9937e8256d45ae9dff11b9b078eaeb" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } } peers { permanent_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } }
I20260928 08:33:37.873059 29556 leader_election.cc:288] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cabf1cf7b2e442f8cc6b5827128f046 (127.15.134.254:41729), fd9937e8256d45ae9dff11b9b078eaeb (127.15.134.253:33329)
I20260928 08:33:37.882042 29394 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } attempt: 4
I20260928 08:33:37.878929 29475 master_runner.cc:186] Error getting master registration for 127.15.134.254:41729: OK, instance_id { permanent_uuid: "5cabf1cf7b2e442f8cc6b5827128f046" instance_seqno: 1790584416738526 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 08:33:37.890758 29475 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 08:33:37.891894 29567 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
Sep 28 08:33:37 dist-test-slave-qp2n krb5kdc[29314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584416, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for kudu/127.15.134.252@KRBTEST.COM
W20260928 08:33:37.892134 29567 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:33:37.892215 29473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.849307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47428 (local address 127.15.134.253:33329)
0928 08:33:37.849501 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:37.849507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:37.858359 (+ 8852us) server_negotiation.cc:407] Connection header received
0928 08:33:37.858663 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:37.858670 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:37.858788 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:37.858900 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:37.861638 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.863224 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.866147 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.866968 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.867088 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:37.872351 (+ 5263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:37.872358 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:37.872365 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:37.872420 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:37.876637 (+ 4217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:37.877912 (+ 1275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:37.877918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:37.877921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:37.878734 (+ 813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:37.879398 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:37.879406 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:37.879410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:37.880234 (+ 824us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:37.880769 (+ 535us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:37.881216 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 08:33:37.881693 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:33:37.901408 29398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.882585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:42687 (local address 127.0.0.1:36538)
0928 08:33:37.882935 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.882955 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:37.883060 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:37.883918 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:37.883924 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:37.884432 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:33:37.884707 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:37.884723 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.886419 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.886427 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:37.887906 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:37.887922 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.889241 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.889249 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:37.889447 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:37.890245 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:33:37.890281 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:33:37.893095 (+ 2814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:33:37.897230 (+ 4135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:37.897240 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:37.897245 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:37.898281 (+ 1036us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:37.898745 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:37.898753 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:37.898757 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:37.899109 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:37.900199 (+ 1090us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:33:37.900206 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:33:37.900383 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 08:33:37.900583 (+ 200us) client_negotiation.cc:240] Negotiation successful
0928 08:33:37.900977 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
I20260928 08:33:37.901423 29545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.883278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36538 (local address 127.15.134.252:42687)
0928 08:33:37.883458 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:37.883464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:37.883480 (+ 16us) server_negotiation.cc:407] Connection header received
0928 08:33:37.883529 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:37.883534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:37.883627 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:37.883729 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:37.884898 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.886195 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.888124 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.888845 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.888990 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:37.893331 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:37.893337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:37.893342 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:37.893384 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:37.897032 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:37.898439 (+ 1407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:37.898444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:37.898447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:37.898588 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:37.899345 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:37.899351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:37.899355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:37.899941 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:37.900075 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:37.900791 (+ 716us) server_negotiation.cc:299] Negotiation successful
0928 08:33:37.901051 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 08:33:37.907747 29394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cabf1cf7b2e442f8cc6b5827128f046: Bootstrap starting.
I20260928 08:33:37.911183 29394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cabf1cf7b2e442f8cc6b5827128f046: Neither blocks nor log segments found. Creating new log.
I20260928 08:33:37.912290 29394 log.cc:824] T 00000000000000000000000000000000 P 5cabf1cf7b2e442f8cc6b5827128f046: Log is configured to *not* fsync() on all Append() calls
I20260928 08:33:37.915864 29394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cabf1cf7b2e442f8cc6b5827128f046: No bootstrap required, opened a new log
I20260928 08:33:37.928567 29394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cabf1cf7b2e442f8cc6b5827128f046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cabf1cf7b2e442f8cc6b5827128f046" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41729 } } peers { permanent_uuid: "fd9937e8256d45ae9dff11b9b078eaeb" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } } peers { permanent_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } }
I20260928 08:33:37.929035 29394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cabf1cf7b2e442f8cc6b5827128f046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:33:37.929180 29394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cabf1cf7b2e442f8cc6b5827128f046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cabf1cf7b2e442f8cc6b5827128f046, State: Initialized, Role: FOLLOWER
I20260928 08:33:37.929912 29394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cabf1cf7b2e442f8cc6b5827128f046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cabf1cf7b2e442f8cc6b5827128f046" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41729 } } peers { permanent_uuid: "fd9937e8256d45ae9dff11b9b078eaeb" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } } peers { permanent_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } }
I20260928 08:33:37.931514 29570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cabf1cf7b2e442f8cc6b5827128f046 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cabf1cf7b2e442f8cc6b5827128f046" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41729 } } peers { permanent_uuid: "fd9937e8256d45ae9dff11b9b078eaeb" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } } peers { permanent_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } } }
I20260928 08:33:37.932515 29570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cabf1cf7b2e442f8cc6b5827128f046 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:33:37.932629 29394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cabf1cf7b2e442f8cc6b5827128f046 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:33:37.932673 29369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cabf1cf7b2e442f8cc6b5827128f046" is_pre_election: true
I20260928 08:33:37.933597 29369 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5cabf1cf7b2e442f8cc6b5827128f046 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd58b2e0defb45bfa5050c52d9fbf8d3 in term 0.
I20260928 08:33:37.934233 29492 leader_election.cc:302] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cabf1cf7b2e442f8cc6b5827128f046, bd58b2e0defb45bfa5050c52d9fbf8d3; no voters:
I20260928 08:33:37.934751 29556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:33:37.934929 29556 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:33:37.935040 29556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:33:37.938802 29556 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cabf1cf7b2e442f8cc6b5827128f046" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41729 } } peers { permanent_uuid: "fd9937e8256d45ae9dff11b9b078eaeb" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } } peers { permanent_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } }
I20260928 08:33:37.939567 29556 leader_election.cc:288] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [CANDIDATE]: Term 1 election: Requested vote from peers 5cabf1cf7b2e442f8cc6b5827128f046 (127.15.134.254:41729), fd9937e8256d45ae9dff11b9b078eaeb (127.15.134.253:33329)
I20260928 08:33:37.940589 29369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cabf1cf7b2e442f8cc6b5827128f046"
I20260928 08:33:37.940909 29369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cabf1cf7b2e442f8cc6b5827128f046 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:33:37.947817 29369 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5cabf1cf7b2e442f8cc6b5827128f046 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd58b2e0defb45bfa5050c52d9fbf8d3 in term 1.
I20260928 08:33:37.948297 29492 leader_election.cc:302] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cabf1cf7b2e442f8cc6b5827128f046, bd58b2e0defb45bfa5050c52d9fbf8d3; no voters:
I20260928 08:33:37.948681 29556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:33:37.948884 29556 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [term 1 LEADER]: Becoming Leader. State: Replica: bd58b2e0defb45bfa5050c52d9fbf8d3, State: Running, Role: LEADER
I20260928 08:33:37.949502 29556 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cabf1cf7b2e442f8cc6b5827128f046" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41729 } } peers { permanent_uuid: "fd9937e8256d45ae9dff11b9b078eaeb" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } } peers { permanent_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } }
I20260928 08:33:37.961491 29555 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd58b2e0defb45bfa5050c52d9fbf8d3. Latest consensus state: current_term: 1 leader_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cabf1cf7b2e442f8cc6b5827128f046" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41729 } } peers { permanent_uuid: "fd9937e8256d45ae9dff11b9b078eaeb" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } } peers { permanent_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } } }
I20260928 08:33:37.961790 29555 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [sys.catalog]: This master's current role is: LEADER
W20260928 08:33:37.964509 29581 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5cabf1cf7b2e442f8cc6b5827128f046: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:33:37.964672 29581 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:33:37.953634 29327 master_runner.cc:186] Error getting master registration for 127.15.134.253:33329: OK, instance_id { permanent_uuid: "fd9937e8256d45ae9dff11b9b078eaeb" instance_seqno: 1790584417246871 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 08:33:37.968294 29327 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 08:33:37.974195 29584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:33:37.976732 29584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260928 08:33:37.977351 29348 tablet.cc:2369] T 00000000000000000000000000000000 P 5cabf1cf7b2e442f8cc6b5827128f046: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:33:37.978199 29468 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } attempt: 5
Sep 28 08:33:37 dist-test-slave-qp2n krb5kdc[29314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584417, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.253@KRBTEST.COM for kudu/127.15.134.252@KRBTEST.COM
W20260928 08:33:37.992410 29496 tablet.cc:2369] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:33:37.994424 29369 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5cabf1cf7b2e442f8cc6b5827128f046 [term 1 FOLLOWER]: Refusing update from remote peer bd58b2e0defb45bfa5050c52d9fbf8d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:33:37.995651 29555 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cabf1cf7b2e442f8cc6b5827128f046" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:33:38.011571 29471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.979214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:42687 (local address 127.0.0.1:36554)
0928 08:33:37.979596 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:37.979645 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:37.979761 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:37.980692 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:37.980699 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:37.981263 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:33:37.981483 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:37.981498 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.983418 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.983425 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:37.985101 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:37.985121 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.986518 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.986524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:37.986788 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:37.987672 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:33:37.987727 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:33:37.992442 (+ 4715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:33:38.003769 (+ 11327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:38.003776 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:38.003781 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:38.004946 (+ 1165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:38.005965 (+ 1019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:38.005971 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:38.006004 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:38.006414 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:38.007882 (+ 1468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:33:38.007918 (+ 36us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:33:38.008217 (+ 299us) client_negotiation.cc:769] Sending connection context
0928 08:33:38.008503 (+ 286us) client_negotiation.cc:240] Negotiation successful
0928 08:33:38.009899 (+ 1396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241}
I20260928 08:33:38.012941 29545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:37.979970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36554 (local address 127.15.134.252:42687)
0928 08:33:37.980155 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:37.980161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:37.980178 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:33:37.980240 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:37.980245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:37.980320 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:37.980445 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:37.981669 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.982806 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.985366 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:37.986101 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:37.986235 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:37.992872 (+ 6637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:37.992881 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:37.992886 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:37.992929 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:38.003481 (+ 10552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:38.005212 (+ 1731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:38.005220 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:38.005226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:38.005767 (+ 541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:38.006602 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:38.006642 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:38.006647 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:38.007644 (+ 997us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:38.009151 (+ 1507us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:38.009829 (+ 678us) server_negotiation.cc:299] Negotiation successful
0928 08:33:38.010934 (+ 1105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 08:33:38.019182 29468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd9937e8256d45ae9dff11b9b078eaeb: Bootstrap starting.
I20260928 08:33:38.022468 29570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cabf1cf7b2e442f8cc6b5827128f046 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd58b2e0defb45bfa5050c52d9fbf8d3. Latest consensus state: current_term: 1 leader_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cabf1cf7b2e442f8cc6b5827128f046" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41729 } } peers { permanent_uuid: "fd9937e8256d45ae9dff11b9b078eaeb" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } } peers { permanent_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } } }
I20260928 08:33:38.022790 29570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cabf1cf7b2e442f8cc6b5827128f046 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:33:38.026527 29555 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cabf1cf7b2e442f8cc6b5827128f046" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41729 } } peers { permanent_uuid: "fd9937e8256d45ae9dff11b9b078eaeb" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } } peers { permanent_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } } }
I20260928 08:33:38.026927 29555 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [sys.catalog]: This master's current role is: LEADER
I20260928 08:33:38.028044 29468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd9937e8256d45ae9dff11b9b078eaeb: Neither blocks nor log segments found. Creating new log.
I20260928 08:33:38.029531 29468 log.cc:824] T 00000000000000000000000000000000 P fd9937e8256d45ae9dff11b9b078eaeb: Log is configured to *not* fsync() on all Append() calls
I20260928 08:33:38.029625 29584 catalog_manager.cc:1347] Generated new cluster ID: 665ead7b5fba4acbb3a2b749b2c294f5
I20260928 08:33:38.029731 29584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:33:38.044293 29468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd9937e8256d45ae9dff11b9b078eaeb: No bootstrap required, opened a new log
I20260928 08:33:38.054356 29570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cabf1cf7b2e442f8cc6b5827128f046 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cabf1cf7b2e442f8cc6b5827128f046" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41729 } } peers { permanent_uuid: "fd9937e8256d45ae9dff11b9b078eaeb" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } } peers { permanent_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } } }
I20260928 08:33:38.054684 29570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cabf1cf7b2e442f8cc6b5827128f046 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:33:38.062857 29468 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd9937e8256d45ae9dff11b9b078eaeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cabf1cf7b2e442f8cc6b5827128f046" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41729 } } peers { permanent_uuid: "fd9937e8256d45ae9dff11b9b078eaeb" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } } peers { permanent_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } }
I20260928 08:33:38.063418 29468 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd9937e8256d45ae9dff11b9b078eaeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:33:38.063587 29468 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd9937e8256d45ae9dff11b9b078eaeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd9937e8256d45ae9dff11b9b078eaeb, State: Initialized, Role: FOLLOWER
I20260928 08:33:38.064438 29468 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd9937e8256d45ae9dff11b9b078eaeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cabf1cf7b2e442f8cc6b5827128f046" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41729 } } peers { permanent_uuid: "fd9937e8256d45ae9dff11b9b078eaeb" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } } peers { permanent_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } }
I20260928 08:33:38.070627 29468 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd9937e8256d45ae9dff11b9b078eaeb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:33:38.071027 29441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd9937e8256d45ae9dff11b9b078eaeb [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:33:38.075446 29441 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd9937e8256d45ae9dff11b9b078eaeb [term 1 FOLLOWER]: Refusing update from remote peer bd58b2e0defb45bfa5050c52d9fbf8d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:33:38.076143 29442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd9937e8256d45ae9dff11b9b078eaeb"
I20260928 08:33:38.076465 29555 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd9937e8256d45ae9dff11b9b078eaeb" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:33:38.077536 29584 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:33:38.079427 29584 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:33:38.080724 29443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd9937e8256d45ae9dff11b9b078eaeb" is_pre_election: true
I20260928 08:33:38.082649 29442 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fd9937e8256d45ae9dff11b9b078eaeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd58b2e0defb45bfa5050c52d9fbf8d3 in term 1.
I20260928 08:33:38.068065 29591 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd9937e8256d45ae9dff11b9b078eaeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cabf1cf7b2e442f8cc6b5827128f046" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41729 } } peers { permanent_uuid: "fd9937e8256d45ae9dff11b9b078eaeb" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } } peers { permanent_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } } }
I20260928 08:33:38.086308 29591 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd9937e8256d45ae9dff11b9b078eaeb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:33:38.110523 29584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3: Generated new TSK 0
I20260928 08:33:38.111979 29584 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:33:38.170058 29591 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd9937e8256d45ae9dff11b9b078eaeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd58b2e0defb45bfa5050c52d9fbf8d3. Latest consensus state: current_term: 1 leader_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cabf1cf7b2e442f8cc6b5827128f046" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41729 } } peers { permanent_uuid: "fd9937e8256d45ae9dff11b9b078eaeb" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } } peers { permanent_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } } }
I20260928 08:33:38.170472 29591 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd9937e8256d45ae9dff11b9b078eaeb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:33:38.170794 29593 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd9937e8256d45ae9dff11b9b078eaeb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cabf1cf7b2e442f8cc6b5827128f046" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41729 } } peers { permanent_uuid: "fd9937e8256d45ae9dff11b9b078eaeb" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } } peers { permanent_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } } }
I20260928 08:33:38.171181 29593 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd9937e8256d45ae9dff11b9b078eaeb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:33:38.172911 29556 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cabf1cf7b2e442f8cc6b5827128f046" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41729 } } peers { permanent_uuid: "fd9937e8256d45ae9dff11b9b078eaeb" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 33329 } } peers { permanent_uuid: "bd58b2e0defb45bfa5050c52d9fbf8d3" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 42687 } } }
I20260928 08:33:38.173303 29556 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd58b2e0defb45bfa5050c52d9fbf8d3 [sys.catalog]: This master's current role is: LEADER
I20260928 08:33:38.205611 29611 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 08:33:38.966814 29581 catalog_manager.cc:1259] Loaded cluster ID: 665ead7b5fba4acbb3a2b749b2c294f5
I20260928 08:33:38.966923 29581 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5cabf1cf7b2e442f8cc6b5827128f046: loading cluster ID for follower catalog manager: success
I20260928 08:33:38.969313 29581 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5cabf1cf7b2e442f8cc6b5827128f046: acquiring CA information for follower catalog manager: success
I20260928 08:33:38.971158 29581 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5cabf1cf7b2e442f8cc6b5827128f046: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:33:39.210109 29611 catalog_manager.cc:1259] Loaded cluster ID: 665ead7b5fba4acbb3a2b749b2c294f5
I20260928 08:33:39.210250 29611 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fd9937e8256d45ae9dff11b9b078eaeb: loading cluster ID for follower catalog manager: success
I20260928 08:33:39.212829 29611 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fd9937e8256d45ae9dff11b9b078eaeb: acquiring CA information for follower catalog manager: success
I20260928 08:33:39.214777 29611 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fd9937e8256d45ae9dff11b9b078eaeb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:33:39.266788 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 29327
I20260928 08:33:39.268141 29389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:33:39.380060 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 29327
I20260928 08:33:39.402266 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 29401
I20260928 08:33:39.403486 29463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:33:39.509191 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 29401
I20260928 08:33:39.526201 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 29475
I20260928 08:33:39.527441 29537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:33:39.637394 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 29475
2026-09-28T08:33:39Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4714 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:33:39 dist-test-slave-qp2n krb5kdc[29618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:33:39 dist-test-slave-qp2n krb5kdc[29618](info): set up 2 sockets
Sep 28 08:33:39 dist-test-slave-qp2n krb5kdc[29618](info): commencing operation
krb5kdc: starting...
W20260928 08:33:40.770574 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:33:40 dist-test-slave-qp2n krb5kdc[29618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:33:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:33:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:33:41.018158 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:34033
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:32777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:34033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:33:41.361644 29631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:33:41.361997 29631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:33:41.362166 29631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:33:41.369463 29631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:33:41.369624 29631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:33:41.369742 29631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:33:41.369848 29631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:33:41.369937 29631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:33:41.379928 29631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:32777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:34033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:34033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:33:41.381361 29631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:33:41.382706 29631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:33:41.395138 29637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:41.395244 29636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:41.395182 29639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:41.396577 29631 server_base.cc:1034] running on GCE node
I20260928 08:33:41.397399 29631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:33:41.398878 29631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:33:41.400110 29631 hybrid_clock.cc:648] HybridClock initialized: now 1790584421400047 us; error 48 us; skew 500 ppm
I20260928 08:33:41.400468 29631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:33:41 dist-test-slave-qp2n krb5kdc[29618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584421, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:33:41.403496 29631 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:33:41.405038 29631 webserver.cc:460] Webserver started at http://127.15.134.254:36857/ using document root <none> and password file <none>
I20260928 08:33:41.405714 29631 fs_manager.cc:359] Metadata directory not provided
I20260928 08:33:41.405865 29631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:33:41.406178 29631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:33:41.409147 29631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "fffc843fbb1f482681dece29d96cc2a0"
format_stamp: "Formatted at 2026-09-28 08:33:41 on dist-test-slave-qp2n"
server_key: "d66a14640ae6872417e11461d9526c65"
server_key_iv: "c45beb7e4bd193ea3de55fb1a16df7aa"
server_key_version: "encryptionkey@0"
I20260928 08:33:41.409929 29631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "fffc843fbb1f482681dece29d96cc2a0"
format_stamp: "Formatted at 2026-09-28 08:33:41 on dist-test-slave-qp2n"
server_key: "d66a14640ae6872417e11461d9526c65"
server_key_iv: "c45beb7e4bd193ea3de55fb1a16df7aa"
server_key_version: "encryptionkey@0"
I20260928 08:33:41.415071 29631 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:33:41.418768 29646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:41.420243 29631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:33:41.420439 29631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "fffc843fbb1f482681dece29d96cc2a0"
format_stamp: "Formatted at 2026-09-28 08:33:41 on dist-test-slave-qp2n"
server_key: "d66a14640ae6872417e11461d9526c65"
server_key_iv: "c45beb7e4bd193ea3de55fb1a16df7aa"
server_key_version: "encryptionkey@0"
I20260928 08:33:41.420665 29631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:33:41.447926 29631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:33:41.452505 29631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:33:41.452860 29631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:33:41.471869 29631 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:34033
I20260928 08:33:41.471900 29697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:34033 every 8 connection(s)
I20260928 08:33:41.473419 29631 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:33:41.477483 29698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:33:41.481613 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 29631
I20260928 08:33:41.481863 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:33:41.482666 15899 external_mini_cluster.cc:1428] Setting key fc403e4e20ccad0e3dcb3e4bf378464f
I20260928 08:33:41.491657 29698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffc843fbb1f482681dece29d96cc2a0: Bootstrap starting.
I20260928 08:33:41.495992 29698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fffc843fbb1f482681dece29d96cc2a0: Neither blocks nor log segments found. Creating new log.
I20260928 08:33:41.497454 29698 log.cc:824] T 00000000000000000000000000000000 P fffc843fbb1f482681dece29d96cc2a0: Log is configured to *not* fsync() on all Append() calls
Sep 28 08:33:41 dist-test-slave-qp2n krb5kdc[29618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:33:41.500509 29698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fffc843fbb1f482681dece29d96cc2a0: No bootstrap required, opened a new log
I20260928 08:33:41.508795 29701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:41.484735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55204 (local address 127.15.134.254:34033)
0928 08:33:41.485555 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:41.485568 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:41.485608 (+ 40us) server_negotiation.cc:407] Connection header received
0928 08:33:41.487246 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:41.487271 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:41.487727 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:41.488077 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:41.490052 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:41.491480 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:41.493465 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:41.494288 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:41.494443 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:41.499765 (+ 5322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:41.499822 (+ 57us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:41.499839 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:41.499881 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:41.503963 (+ 4082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:41.504826 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:41.504845 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:41.504855 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:41.505075 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:41.505510 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:41.505515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:41.505518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:41.506244 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:41.506555 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:41.506894 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 08:33:41.507356 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":379,"threads_started":1}
I20260928 08:33:41.511349 29698 raft_consensus.cc:348] T 00000000000000000000000000000000 P fffc843fbb1f482681dece29d96cc2a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffc843fbb1f482681dece29d96cc2a0" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 34033 } }
I20260928 08:33:41.511754 29698 raft_consensus.cc:374] T 00000000000000000000000000000000 P fffc843fbb1f482681dece29d96cc2a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:33:41.511865 29698 raft_consensus.cc:729] T 00000000000000000000000000000000 P fffc843fbb1f482681dece29d96cc2a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fffc843fbb1f482681dece29d96cc2a0, State: Initialized, Role: FOLLOWER
I20260928 08:33:41.512487 29698 consensus_queue.cc:260] T 00000000000000000000000000000000 P fffc843fbb1f482681dece29d96cc2a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffc843fbb1f482681dece29d96cc2a0" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 34033 } }
I20260928 08:33:41.512809 29698 raft_consensus.cc:388] T 00000000000000000000000000000000 P fffc843fbb1f482681dece29d96cc2a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:33:41.512985 29698 raft_consensus.cc:482] T 00000000000000000000000000000000 P fffc843fbb1f482681dece29d96cc2a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:33:41.513105 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:33:41.513242 29698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fffc843fbb1f482681dece29d96cc2a0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:33:41.515539 29698 raft_consensus.cc:504] T 00000000000000000000000000000000 P fffc843fbb1f482681dece29d96cc2a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffc843fbb1f482681dece29d96cc2a0" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 34033 } }
I20260928 08:33:41.516112 29698 leader_election.cc:302] T 00000000000000000000000000000000 P fffc843fbb1f482681dece29d96cc2a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fffc843fbb1f482681dece29d96cc2a0; no voters:
I20260928 08:33:41.516806 29698 leader_election.cc:288] T 00000000000000000000000000000000 P fffc843fbb1f482681dece29d96cc2a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:33:41.516957 29703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fffc843fbb1f482681dece29d96cc2a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:33:41.517632 29703 raft_consensus.cc:686] T 00000000000000000000000000000000 P fffc843fbb1f482681dece29d96cc2a0 [term 1 LEADER]: Becoming Leader. State: Replica: fffc843fbb1f482681dece29d96cc2a0, State: Running, Role: LEADER
I20260928 08:33:41.518200 29703 consensus_queue.cc:237] T 00000000000000000000000000000000 P fffc843fbb1f482681dece29d96cc2a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffc843fbb1f482681dece29d96cc2a0" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 34033 } }
I20260928 08:33:41.518865 29698 sys_catalog.cc:564] T 00000000000000000000000000000000 P fffc843fbb1f482681dece29d96cc2a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:33:41.520941 29704 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffc843fbb1f482681dece29d96cc2a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fffc843fbb1f482681dece29d96cc2a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffc843fbb1f482681dece29d96cc2a0" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 34033 } } }
I20260928 08:33:41.521373 29704 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffc843fbb1f482681dece29d96cc2a0 [sys.catalog]: This master's current role is: LEADER
I20260928 08:33:41.521979 29705 sys_catalog.cc:455] T 00000000000000000000000000000000 P fffc843fbb1f482681dece29d96cc2a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fffc843fbb1f482681dece29d96cc2a0. Latest consensus state: current_term: 1 leader_uuid: "fffc843fbb1f482681dece29d96cc2a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fffc843fbb1f482681dece29d96cc2a0" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 34033 } } }
I20260928 08:33:41.522282 29705 sys_catalog.cc:458] T 00000000000000000000000000000000 P fffc843fbb1f482681dece29d96cc2a0 [sys.catalog]: This master's current role is: LEADER
I20260928 08:33:41.523037 29709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:33:41.528492 29709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:33:41.538846 29709 catalog_manager.cc:1347] Generated new cluster ID: d8ac4d2e9e9c487abd3602fd6a702fb3
I20260928 08:33:41.538976 29709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:33:41.555536 29709 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:33:41.557026 29709 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:33:41.565433 29709 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fffc843fbb1f482681dece29d96cc2a0: Generated new TSK 0
I20260928 08:33:41.566325 29709 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:33:41.625528 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 29631
I20260928 08:33:41.626721 29693 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:33:41.726406 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 29631
2026-09-28T08:33:41Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2084 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:33:41 dist-test-slave-qp2n krb5kdc[29726](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:33:41 dist-test-slave-qp2n krb5kdc[29726](info): set up 2 sockets
Sep 28 08:33:41 dist-test-slave-qp2n krb5kdc[29726](info): commencing operation
krb5kdc: starting...
W20260928 08:33:42.854642 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:33:43 dist-test-slave-qp2n krb5kdc[29726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:33:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:33:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:33:43.101476 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:41481
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:44545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:41481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:33:43.460206 29739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:33:43.460511 29739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:33:43.460601 29739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:33:43.468046 29739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:33:43.468160 29739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:33:43.468206 29739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:33:43.468257 29739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:33:43.468302 29739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:33:43.478624 29739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:44545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:41481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:41481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:33:43.479967 29739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:33:43.481348 29739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:33:43.494122 29745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:43.494266 29747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:43.494187 29744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:43.494943 29739 server_base.cc:1034] running on GCE node
I20260928 08:33:43.495831 29739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:33:43.497483 29739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:33:43.498739 29739 hybrid_clock.cc:648] HybridClock initialized: now 1790584423498678 us; error 51 us; skew 500 ppm
I20260928 08:33:43.499130 29739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:33:43 dist-test-slave-qp2n krb5kdc[29726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584423, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:33:43.502537 29739 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:33:43.504256 29739 webserver.cc:460] Webserver started at http://127.15.134.254:42229/ using document root <none> and password file <none>
I20260928 08:33:43.505127 29739 fs_manager.cc:359] Metadata directory not provided
I20260928 08:33:43.505288 29739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:33:43.505620 29739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:33:43.508967 29739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "16349924272d46a988a8e12fa7eb8360"
format_stamp: "Formatted at 2026-09-28 08:33:43 on dist-test-slave-qp2n"
server_key: "3fa15451eb890da685b9562e5ec9303c"
server_key_iv: "56738fcd9f8f0b7460f5fb5a07e5e2d1"
server_key_version: "encryptionkey@0"
I20260928 08:33:43.509790 29739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "16349924272d46a988a8e12fa7eb8360"
format_stamp: "Formatted at 2026-09-28 08:33:43 on dist-test-slave-qp2n"
server_key: "3fa15451eb890da685b9562e5ec9303c"
server_key_iv: "56738fcd9f8f0b7460f5fb5a07e5e2d1"
server_key_version: "encryptionkey@0"
I20260928 08:33:43.515111 29739 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 08:33:43.518860 29754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:43.520283 29739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:33:43.520431 29739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "16349924272d46a988a8e12fa7eb8360"
format_stamp: "Formatted at 2026-09-28 08:33:43 on dist-test-slave-qp2n"
server_key: "3fa15451eb890da685b9562e5ec9303c"
server_key_iv: "56738fcd9f8f0b7460f5fb5a07e5e2d1"
server_key_version: "encryptionkey@0"
I20260928 08:33:43.520596 29739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:33:43.534500 29739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:33:43.538919 29739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:33:43.539155 29739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:33:43.558555 29739 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:41481
I20260928 08:33:43.558589 29805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:41481 every 8 connection(s)
I20260928 08:33:43.560037 29739 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:33:43.563911 29806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:33:43.564957 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 29739
I20260928 08:33:43.565192 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:33:43.566068 15899 external_mini_cluster.cc:1428] Setting key 158b7e7bc1a3278caf937c0474e31a16
I20260928 08:33:43.578816 29806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360: Bootstrap starting.
Sep 28 08:33:43 dist-test-slave-qp2n krb5kdc[29726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:33:43.582063 29806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360: Neither blocks nor log segments found. Creating new log.
I20260928 08:33:43.583479 29806 log.cc:824] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360: Log is configured to *not* fsync() on all Append() calls
I20260928 08:33:43.586880 29806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360: No bootstrap required, opened a new log
I20260928 08:33:43.591473 29809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:43.568072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52300 (local address 127.15.134.254:41481)
0928 08:33:43.568837 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:43.568857 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:43.568912 (+ 55us) server_negotiation.cc:407] Connection header received
0928 08:33:43.571071 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:43.571099 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:43.571612 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:43.572044 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:43.573385 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:43.575381 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:43.576764 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:43.577809 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:43.577994 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:43.582280 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:43.582291 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:43.582312 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:43.582359 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:43.586440 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:43.587215 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:43.587223 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:43.587231 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:43.587465 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:43.587880 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:43.587885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:43.587888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:43.588580 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:43.588958 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:43.589616 (+ 658us) server_negotiation.cc:299] Negotiation successful
0928 08:33:43.590084 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":283,"threads_started":1}
I20260928 08:33:43.596673 29806 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16349924272d46a988a8e12fa7eb8360" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41481 } }
I20260928 08:33:43.597036 29806 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:33:43.597119 29806 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16349924272d46a988a8e12fa7eb8360, State: Initialized, Role: FOLLOWER
I20260928 08:33:43.597676 29806 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16349924272d46a988a8e12fa7eb8360" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41481 } }
I20260928 08:33:43.597887 29806 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:33:43.597985 29806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:33:43.598100 29806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:33:43.600242 29806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16349924272d46a988a8e12fa7eb8360" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41481 } }
I20260928 08:33:43.600914 29806 leader_election.cc:302] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16349924272d46a988a8e12fa7eb8360; no voters:
I20260928 08:33:43.601509 29806 leader_election.cc:288] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:33:43.601650 29811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:33:43.602334 29811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [term 1 LEADER]: Becoming Leader. State: Replica: 16349924272d46a988a8e12fa7eb8360, State: Running, Role: LEADER
I20260928 08:33:43.602891 29811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16349924272d46a988a8e12fa7eb8360" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41481 } }
I20260928 08:33:43.603609 29806 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:33:43.605942 29812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16349924272d46a988a8e12fa7eb8360" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16349924272d46a988a8e12fa7eb8360" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41481 } } }
I20260928 08:33:43.605918 29813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16349924272d46a988a8e12fa7eb8360. Latest consensus state: current_term: 1 leader_uuid: "16349924272d46a988a8e12fa7eb8360" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16349924272d46a988a8e12fa7eb8360" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41481 } } }
I20260928 08:33:43.606401 29812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [sys.catalog]: This master's current role is: LEADER
I20260928 08:33:43.606402 29813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [sys.catalog]: This master's current role is: LEADER
I20260928 08:33:43.613803 29816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:33:43.618588 29816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:33:43.627475 29816 catalog_manager.cc:1347] Generated new cluster ID: 0aaa7967c25b48dcbfdfa9ac3df6232e
I20260928 08:33:43.627599 29816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:33:43.651919 29816 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:33:43.653491 29816 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:33:43.668550 29816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360: Generated new TSK 0
I20260928 08:33:43.669569 29816 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.134.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
I20260928 08:33:43.735703 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:0
--local_ip_for_outbound_sockets=127.15.134.193
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:41481
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:44545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:33:44.094760 29832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:33:44.095052 29832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:33:44.095206 29832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:33:44.102746 29832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:33:44.102864 29832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:33:44.103049 29832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:33:44.113315 29832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:44545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:41481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:33:44.114676 29832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:33:44.116096 29832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:33:44.129846 29837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:44.129976 29840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:44.129987 29838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:44.130638 29832 server_base.cc:1034] running on GCE node
I20260928 08:33:44.131268 29832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:33:44.132328 29832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:33:44.133531 29832 hybrid_clock.cc:648] HybridClock initialized: now 1790584424133470 us; error 61 us; skew 500 ppm
I20260928 08:33:44.133874 29832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:33:44 dist-test-slave-qp2n krb5kdc[29726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584424, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:33:44.137231 29832 init.cc:377] Logged in from keytab as kudu/127.15.134.193@KRBTEST.COM (short username kudu)
I20260928 08:33:44.138862 29832 webserver.cc:460] Webserver started at http://127.15.134.193:40457/ using document root <none> and password file <none>
I20260928 08:33:44.139570 29832 fs_manager.cc:359] Metadata directory not provided
I20260928 08:33:44.139722 29832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:33:44.140044 29832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:33:44.143137 29832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/data/instance:
uuid: "28cdcfc4d9d14ccd91005a973261bfac"
format_stamp: "Formatted at 2026-09-28 08:33:44 on dist-test-slave-qp2n"
server_key: "2ea04a7635602a12c19de0ca73dd8f59"
server_key_iv: "930d0735e018f10a56fb8bc660685917"
server_key_version: "encryptionkey@0"
I20260928 08:33:44.144012 29832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance:
uuid: "28cdcfc4d9d14ccd91005a973261bfac"
format_stamp: "Formatted at 2026-09-28 08:33:44 on dist-test-slave-qp2n"
server_key: "2ea04a7635602a12c19de0ca73dd8f59"
server_key_iv: "930d0735e018f10a56fb8bc660685917"
server_key_version: "encryptionkey@0"
I20260928 08:33:44.149828 29832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260928 08:33:44.153762 29847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:44.155282 29832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260928 08:33:44.155472 29832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "28cdcfc4d9d14ccd91005a973261bfac"
format_stamp: "Formatted at 2026-09-28 08:33:44 on dist-test-slave-qp2n"
server_key: "2ea04a7635602a12c19de0ca73dd8f59"
server_key_iv: "930d0735e018f10a56fb8bc660685917"
server_key_version: "encryptionkey@0"
I20260928 08:33:44.155694 29832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:33:44.193420 29832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:33:44.197902 29832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:33:44.198217 29832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:33:44.199308 29832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:33:44.200760 29832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:33:44.200886 29832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:44.201040 29832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:33:44.201114 29832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:44.240145 29832 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:33821
I20260928 08:33:44.240190 29959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:33821 every 8 connection(s)
I20260928 08:33:44.241554 29832 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:33:44.244728 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 29832
I20260928 08:33:44.244956 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance
I20260928 08:33:44.245850 15899 external_mini_cluster.cc:1428] Setting key 048a605c1f4a0038ebb7cae059f7a573
Sep 28 08:33:44 dist-test-slave-qp2n krb5kdc[29726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584424, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:33:44.266178 29962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:44.244193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:43045 (local address 127.15.134.254:41481)
0928 08:33:44.244755 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:44.244761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:44.245872 (+ 1111us) server_negotiation.cc:407] Connection header received
0928 08:33:44.248248 (+ 2376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:44.248252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:44.248319 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:44.248418 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:44.250681 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:44.251416 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:44.254022 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:44.254429 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:44.254529 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:44.258980 (+ 4451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:44.258989 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:44.258997 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:44.259045 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:44.262745 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:44.263509 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:44.263517 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:44.263523 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:44.263681 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:44.264102 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:44.264109 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:44.264115 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:44.264511 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:44.264696 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:44.265425 (+ 729us) server_negotiation.cc:299] Negotiation successful
0928 08:33:44.265687 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":257,"threads_started":1}
I20260928 08:33:44.266873 29963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:44.244747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.193:43045)
0928 08:33:44.245668 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:44.245733 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:44.247922 (+ 2189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:44.248725 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:44.248738 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:44.249573 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:33:44.250466 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:44.250501 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:44.251591 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:44.251600 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:44.253868 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:44.253887 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:44.254668 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:44.254677 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:44.255001 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:44.255775 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:33:44.255811 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:33:44.258687 (+ 2876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:33:44.262909 (+ 4222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:44.262917 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:44.262930 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:44.263370 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:44.263794 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:44.263798 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:44.263800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:44.263977 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:44.264685 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:33:44.264692 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:33:44.265029 (+ 337us) client_negotiation.cc:769] Sending connection context
0928 08:33:44.265264 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 08:33:44.265681 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":326,"threads_started":1}
I20260928 08:33:44.268533 29960 heartbeater.cc:347] Connected to a master server at 127.15.134.254:41481
I20260928 08:33:44.268903 29960 heartbeater.cc:464] Registering TS with master...
I20260928 08:33:44.269925 29960 heartbeater.cc:511] Master 127.15.134.254:41481 requested a full tablet report, sending...
I20260928 08:33:44.272421 29771 ts_manager.cc:194] Registered new tserver with Master: 28cdcfc4d9d14ccd91005a973261bfac (127.15.134.193:33821)
WARNING: no policy specified for kudu/127.15.134.194@KRBTEST.COM; defaulting to no policy
I20260928 08:33:44.274040 29771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.193@KRBTEST.COM'} at 127.15.134.193:43045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
I20260928 08:33:44.304876 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:0
--local_ip_for_outbound_sockets=127.15.134.194
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:41481
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:44545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:33:44.654109 29966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:33:44.654438 29966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:33:44.654539 29966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:33:44.662231 29966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:33:44.662361 29966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:33:44.662477 29966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:33:44.672684 29966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:44545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:41481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:33:44.674005 29966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:33:44.675359 29966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:33:44.690207 29972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:44.690217 29971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:44.690632 29974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:44.692217 29966 server_base.cc:1034] running on GCE node
I20260928 08:33:44.693041 29966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:33:44.694150 29966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:33:44.695442 29966 hybrid_clock.cc:648] HybridClock initialized: now 1790584424695404 us; error 51 us; skew 500 ppm
I20260928 08:33:44.695804 29966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:33:44 dist-test-slave-qp2n krb5kdc[29726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584424, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:33:44.699491 29966 init.cc:377] Logged in from keytab as kudu/127.15.134.194@KRBTEST.COM (short username kudu)
I20260928 08:33:44.701398 29966 webserver.cc:460] Webserver started at http://127.15.134.194:36563/ using document root <none> and password file <none>
I20260928 08:33:44.702135 29966 fs_manager.cc:359] Metadata directory not provided
I20260928 08:33:44.702291 29966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:33:44.702720 29966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:33:44.706008 29966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/data/instance:
uuid: "84a23a8f68e047458b7932ff1803136b"
format_stamp: "Formatted at 2026-09-28 08:33:44 on dist-test-slave-qp2n"
server_key: "788e99bc59a6ee4d7bec42261b4a3fb2"
server_key_iv: "b3cc99c786929eccc3217f1290c8d219"
server_key_version: "encryptionkey@0"
I20260928 08:33:44.706841 29966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance:
uuid: "84a23a8f68e047458b7932ff1803136b"
format_stamp: "Formatted at 2026-09-28 08:33:44 on dist-test-slave-qp2n"
server_key: "788e99bc59a6ee4d7bec42261b4a3fb2"
server_key_iv: "b3cc99c786929eccc3217f1290c8d219"
server_key_version: "encryptionkey@0"
I20260928 08:33:44.712219 29966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 08:33:44.715975 29981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:44.717465 29966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:33:44.717649 29966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "84a23a8f68e047458b7932ff1803136b"
format_stamp: "Formatted at 2026-09-28 08:33:44 on dist-test-slave-qp2n"
server_key: "788e99bc59a6ee4d7bec42261b4a3fb2"
server_key_iv: "b3cc99c786929eccc3217f1290c8d219"
server_key_version: "encryptionkey@0"
I20260928 08:33:44.717871 29966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:33:44.736528 29966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:33:44.740974 29966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:33:44.741302 29966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:33:44.742339 29966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:33:44.743813 29966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:33:44.743924 29966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:44.744083 29966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:33:44.744163 29966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:44.780239 29966 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:46523
I20260928 08:33:44.780333 30093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:46523 every 8 connection(s)
I20260928 08:33:44.781697 29966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:33:44.789541 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 29966
I20260928 08:33:44.789763 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance
I20260928 08:33:44.790257 15899 external_mini_cluster.cc:1428] Setting key 52a4b396738cc46751c6680c31601598
Sep 28 08:33:44 dist-test-slave-qp2n krb5kdc[29726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584424, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:33:44.805680 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:44.784433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:47963 (local address 127.15.134.254:41481)
0928 08:33:44.784921 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:44.784926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:44.785926 (+ 1000us) server_negotiation.cc:407] Connection header received
0928 08:33:44.787793 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:44.787798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:44.787895 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:44.788026 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:44.789688 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:44.790518 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:44.792293 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:44.793106 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:44.793275 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:44.797459 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:44.797470 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:44.797478 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:44.797532 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:44.801450 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:44.802753 (+ 1303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:44.802758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:44.802760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:44.802913 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:44.803456 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:44.803460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:44.803462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:44.804022 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:44.804164 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:44.805193 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0928 08:33:44.805378 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":275,"threads_started":1}
I20260928 08:33:44.807186 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:44.785016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.194:47963)
0928 08:33:44.785746 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:44.785804 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:44.787496 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:44.788275 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:44.788284 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:44.788901 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:33:44.789508 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:44.789526 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:44.790681 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:44.790685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:44.792095 (+ 1410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:44.792134 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:44.793341 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:44.793346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:44.793577 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:44.794440 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:33:44.794464 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:33:44.797168 (+ 2704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:33:44.801653 (+ 4485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:44.801667 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:44.801689 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:44.802456 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:44.803045 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:44.803053 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:44.803059 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:44.803327 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:44.804175 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:33:44.804187 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:33:44.804740 (+ 553us) client_negotiation.cc:769] Sending connection context
0928 08:33:44.805071 (+ 331us) client_negotiation.cc:240] Negotiation successful
0928 08:33:44.805694 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":292,"threads_started":1}
I20260928 08:33:44.808776 30094 heartbeater.cc:347] Connected to a master server at 127.15.134.254:41481
I20260928 08:33:44.809095 30094 heartbeater.cc:464] Registering TS with master...
I20260928 08:33:44.809907 30094 heartbeater.cc:511] Master 127.15.134.254:41481 requested a full tablet report, sending...
I20260928 08:33:44.811398 29771 ts_manager.cc:194] Registered new tserver with Master: 84a23a8f68e047458b7932ff1803136b (127.15.134.194:46523)
I20260928 08:33:44.812247 29771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.194@KRBTEST.COM'} at 127.15.134.194:47963
WARNING: no policy specified for kudu/127.15.134.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
I20260928 08:33:44.847486 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:0
--local_ip_for_outbound_sockets=127.15.134.195
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:41481
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:44545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:33:45.199014 30100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:33:45.199393 30100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:33:45.199570 30100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:33:45.207844 30100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:33:45.207996 30100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:33:45.208176 30100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:33:45.219583 30100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:44545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:41481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:33:45.221119 30100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:33:45.222537 30100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:33:45.236440 30106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:45.236442 30108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:45.237282 30100 server_base.cc:1034] running on GCE node
W20260928 08:33:45.237308 30105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:45.238019 30100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:33:45.238992 30100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:33:45.240386 30100 hybrid_clock.cc:648] HybridClock initialized: now 1790584425240351 us; error 58 us; skew 500 ppm
I20260928 08:33:45.240881 30100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:33:45 dist-test-slave-qp2n krb5kdc[29726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584425, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:33:45.244542 30100 init.cc:377] Logged in from keytab as kudu/127.15.134.195@KRBTEST.COM (short username kudu)
I20260928 08:33:45.246096 30100 webserver.cc:460] Webserver started at http://127.15.134.195:34823/ using document root <none> and password file <none>
I20260928 08:33:45.246865 30100 fs_manager.cc:359] Metadata directory not provided
I20260928 08:33:45.247016 30100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:33:45.247352 30100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:33:45.250810 30100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/data/instance:
uuid: "f996423f570146bdb87f10a69d734d2c"
format_stamp: "Formatted at 2026-09-28 08:33:45 on dist-test-slave-qp2n"
server_key: "397a852c371f5c109bda49e7d3b9ef39"
server_key_iv: "74a5e53a3c35c6f34c1b47333d20504e"
server_key_version: "encryptionkey@0"
I20260928 08:33:45.251708 30100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance:
uuid: "f996423f570146bdb87f10a69d734d2c"
format_stamp: "Formatted at 2026-09-28 08:33:45 on dist-test-slave-qp2n"
server_key: "397a852c371f5c109bda49e7d3b9ef39"
server_key_iv: "74a5e53a3c35c6f34c1b47333d20504e"
server_key_version: "encryptionkey@0"
I20260928 08:33:45.257747 30100 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 08:33:45.262038 30115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:45.263602 30100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260928 08:33:45.263798 30100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "f996423f570146bdb87f10a69d734d2c"
format_stamp: "Formatted at 2026-09-28 08:33:45 on dist-test-slave-qp2n"
server_key: "397a852c371f5c109bda49e7d3b9ef39"
server_key_iv: "74a5e53a3c35c6f34c1b47333d20504e"
server_key_version: "encryptionkey@0"
I20260928 08:33:45.264047 30100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:33:45.277436 29960 heartbeater.cc:503] Master 127.15.134.254:41481 was elected leader, sending a full tablet report...
I20260928 08:33:45.281945 30100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:33:45.286441 30100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:33:45.286749 30100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:33:45.287753 30100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:33:45.289170 30100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:33:45.289269 30100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:45.289427 30100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:33:45.289501 30100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:45.323786 30100 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:44605
I20260928 08:33:45.323809 30227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:44605 every 8 connection(s)
I20260928 08:33:45.325292 30100 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:33:45.332062 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 30100
I20260928 08:33:45.332262 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance
I20260928 08:33:45.332803 15899 external_mini_cluster.cc:1428] Setting key 1350af061d35763ab1f063cdf993c513
Sep 28 08:33:45 dist-test-slave-qp2n krb5kdc[29726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584425, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:33:45.346889 30230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:45.327976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:37859 (local address 127.15.134.254:41481)
0928 08:33:45.328519 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:45.328524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:45.329629 (+ 1105us) server_negotiation.cc:407] Connection header received
0928 08:33:45.331410 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:45.331414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:45.331483 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:45.331585 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:45.333484 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.334539 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.336112 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.336544 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.336754 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:45.340290 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:45.340296 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:45.340301 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:45.340336 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:45.343279 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:45.344126 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:45.344135 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:45.344141 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:45.344300 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:45.344853 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:45.344861 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:45.344864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:45.345175 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:45.345348 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:45.346155 (+ 807us) server_negotiation.cc:299] Negotiation successful
0928 08:33:45.346419 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":289,"threads_started":1}
I20260928 08:33:45.347949 30231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:45.328599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.195:37859)
0928 08:33:45.329468 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:45.329508 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:45.331135 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:45.331859 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:45.331867 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:45.332553 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:33:45.333263 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:45.333282 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.334710 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.334715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:45.335956 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:45.335968 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.336844 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.336851 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:45.337106 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:45.337773 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:33:45.337797 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:33:45.340063 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:33:45.343473 (+ 3410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:45.343483 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:45.343505 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:45.343983 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:45.344427 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:45.344436 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:45.344441 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:45.344732 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:45.345330 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:33:45.345339 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:33:45.345730 (+ 391us) client_negotiation.cc:769] Sending connection context
0928 08:33:45.346042 (+ 312us) client_negotiation.cc:240] Negotiation successful
0928 08:33:45.346629 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":301,"threads_started":1}
I20260928 08:33:45.349681 30228 heartbeater.cc:347] Connected to a master server at 127.15.134.254:41481
I20260928 08:33:45.350021 30228 heartbeater.cc:464] Registering TS with master...
I20260928 08:33:45.350844 30228 heartbeater.cc:511] Master 127.15.134.254:41481 requested a full tablet report, sending...
I20260928 08:33:45.352466 29771 ts_manager.cc:194] Registered new tserver with Master: f996423f570146bdb87f10a69d734d2c (127.15.134.195:44605)
I20260928 08:33:45.353544 29771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.195@KRBTEST.COM'} at 127.15.134.195:37859
I20260928 08:33:45.360925 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:33:45.378566 30230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:45.365629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52306 (local address 127.15.134.254:41481)
0928 08:33:45.365826 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:45.365831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:45.366276 (+ 445us) server_negotiation.cc:407] Connection header received
0928 08:33:45.366418 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:45.366422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:45.366485 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:45.366568 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:45.367930 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.368774 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.370324 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.370713 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.370830 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:45.373419 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:45.373425 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:45.373428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:45.373462 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:45.376070 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:45.376762 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:45.376766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:45.376768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:45.376908 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:45.377275 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:45.377278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:45.377281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:45.377592 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:45.377774 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:45.378152 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 08:33:45.378334 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 08:33:45.383188 29771 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:33:45.390391 29771 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:33:45.414846 30244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:45.403706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59880 (local address 127.15.134.194:46523)
0928 08:33:45.405643 (+ 1937us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:45.405653 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:45.406361 (+ 708us) server_negotiation.cc:407] Connection header received
0928 08:33:45.406484 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:45.406493 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:45.406765 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:45.407473 (+ 708us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:33:45.408066 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.409892 (+ 1826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.412460 (+ 2568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.413702 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.413893 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:45.414074 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:45.414347 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 08:33:45.414536 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1750,"thread_start_us":1637,"threads_started":1}
I20260928 08:33:45.415486 30241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:45.403069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:33821 (local address 127.0.0.1:52996)
0928 08:33:45.404683 (+ 1614us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:45.404738 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:45.404976 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:45.407029 (+ 2053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:45.407038 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:45.407083 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:33:45.407721 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:45.407734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.409716 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.409724 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:45.412427 (+ 2703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:45.412443 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.414574 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.414581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:45.414799 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:45.414838 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 08:33:45.414919 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 08:33:45.415067 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1169,"mutex_wait_us":55,"spinlock_wait_cycles":896,"thread_start_us":962,"threads_started":1}
I20260928 08:33:45.415098 30243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:45.403462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:46523 (local address 127.0.0.1:59880)
0928 08:33:45.406123 (+ 2661us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:45.406145 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:45.406313 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:45.407029 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:45.407038 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:45.407090 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:33:45.407840 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:45.407855 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.410041 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.410047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:45.412272 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:45.412284 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.413898 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.413904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:45.414087 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:45.414147 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 08:33:45.414297 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 08:33:45.414584 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2476,"mutex_wait_us":126,"spinlock_wait_cycles":9216,"thread_start_us":2349,"threads_started":1}
I20260928 08:33:45.416118 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:45.403393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52996 (local address 127.15.134.193:33821)
0928 08:33:45.406410 (+ 3017us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:45.406419 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:45.406440 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:33:45.406533 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:45.406539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:45.406765 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:45.407957 (+ 1192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:33:45.408244 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.409581 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.412649 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.414275 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.415275 (+ 1000us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:45.415468 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:45.415549 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 08:33:45.415719 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2807,"thread_start_us":2635,"threads_started":1}
I20260928 08:33:45.416153 30245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:45.406262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:44605 (local address 127.0.0.1:53432)
0928 08:33:45.406896 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:45.406914 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:45.407034 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:45.408300 (+ 1266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:45.408305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:45.408334 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:33:45.408839 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:45.408854 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.410839 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.410846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:45.413203 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:45.413218 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.415275 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.415282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:45.415466 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:45.415508 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 08:33:45.415581 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 08:33:45.415714 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":1,"thread_start_us":794,"threads_started":1}
I20260928 08:33:45.417333 30246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:45.406913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53432 (local address 127.15.134.195:44605)
0928 08:33:45.407620 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:45.407631 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:45.407669 (+ 38us) server_negotiation.cc:407] Connection header received
0928 08:33:45.407778 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:45.407788 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:45.408055 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:45.409040 (+ 985us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:33:45.409335 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.410665 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.413443 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.415029 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.416476 (+ 1447us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:45.416708 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:45.416784 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 08:33:45.416972 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":378,"threads_started":1}
I20260928 08:33:45.419726 30029 tablet_service.cc:1467] Processing CreateTablet for tablet 0fdf1988b86e4d81904fe880bd777beb (DEFAULT_TABLE table=test-table [id=bd42f379d4c34fb19b715a55cd68de10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:33:45.421216 30029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fdf1988b86e4d81904fe880bd777beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:33:45.420868 30163 tablet_service.cc:1467] Processing CreateTablet for tablet 0fdf1988b86e4d81904fe880bd777beb (DEFAULT_TABLE table=test-table [id=bd42f379d4c34fb19b715a55cd68de10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:33:45.420763 29895 tablet_service.cc:1467] Processing CreateTablet for tablet 0fdf1988b86e4d81904fe880bd777beb (DEFAULT_TABLE table=test-table [id=bd42f379d4c34fb19b715a55cd68de10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:33:45.422382 30163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fdf1988b86e4d81904fe880bd777beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:33:45.422375 29895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fdf1988b86e4d81904fe880bd777beb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:33:45.433591 30247 tablet_bootstrap.cc:492] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b: Bootstrap starting.
I20260928 08:33:45.438094 30247 tablet_bootstrap.cc:654] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b: Neither blocks nor log segments found. Creating new log.
I20260928 08:33:45.438376 30248 tablet_bootstrap.cc:492] T 0fdf1988b86e4d81904fe880bd777beb P 28cdcfc4d9d14ccd91005a973261bfac: Bootstrap starting.
I20260928 08:33:45.440192 30249 tablet_bootstrap.cc:492] T 0fdf1988b86e4d81904fe880bd777beb P f996423f570146bdb87f10a69d734d2c: Bootstrap starting.
I20260928 08:33:45.441036 30247 log.cc:824] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:33:45.442328 30248 tablet_bootstrap.cc:654] T 0fdf1988b86e4d81904fe880bd777beb P 28cdcfc4d9d14ccd91005a973261bfac: Neither blocks nor log segments found. Creating new log.
I20260928 08:33:45.444003 30249 tablet_bootstrap.cc:654] T 0fdf1988b86e4d81904fe880bd777beb P f996423f570146bdb87f10a69d734d2c: Neither blocks nor log segments found. Creating new log.
I20260928 08:33:45.444031 30248 log.cc:824] T 0fdf1988b86e4d81904fe880bd777beb P 28cdcfc4d9d14ccd91005a973261bfac: Log is configured to *not* fsync() on all Append() calls
I20260928 08:33:45.444790 30247 tablet_bootstrap.cc:492] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b: No bootstrap required, opened a new log
I20260928 08:33:45.445084 30247 ts_tablet_manager.cc:1397] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.006s
I20260928 08:33:45.445449 30249 log.cc:824] T 0fdf1988b86e4d81904fe880bd777beb P f996423f570146bdb87f10a69d734d2c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:33:45.447292 30248 tablet_bootstrap.cc:492] T 0fdf1988b86e4d81904fe880bd777beb P 28cdcfc4d9d14ccd91005a973261bfac: No bootstrap required, opened a new log
I20260928 08:33:45.447554 30248 ts_tablet_manager.cc:1397] T 0fdf1988b86e4d81904fe880bd777beb P 28cdcfc4d9d14ccd91005a973261bfac: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260928 08:33:45.448190 30249 tablet_bootstrap.cc:492] T 0fdf1988b86e4d81904fe880bd777beb P f996423f570146bdb87f10a69d734d2c: No bootstrap required, opened a new log
I20260928 08:33:45.448460 30249 ts_tablet_manager.cc:1397] T 0fdf1988b86e4d81904fe880bd777beb P f996423f570146bdb87f10a69d734d2c: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260928 08:33:45.455555 30247 raft_consensus.cc:348] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } } peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } }
I20260928 08:33:45.456003 30247 raft_consensus.cc:374] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:33:45.456147 30247 raft_consensus.cc:729] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84a23a8f68e047458b7932ff1803136b, State: Initialized, Role: FOLLOWER
I20260928 08:33:45.456657 30249 raft_consensus.cc:348] T 0fdf1988b86e4d81904fe880bd777beb P f996423f570146bdb87f10a69d734d2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } } peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } }
I20260928 08:33:45.456813 30247 consensus_queue.cc:260] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } } peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } }
I20260928 08:33:45.457053 30249 raft_consensus.cc:374] T 0fdf1988b86e4d81904fe880bd777beb P f996423f570146bdb87f10a69d734d2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:33:45.457176 30249 raft_consensus.cc:729] T 0fdf1988b86e4d81904fe880bd777beb P f996423f570146bdb87f10a69d734d2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f996423f570146bdb87f10a69d734d2c, State: Initialized, Role: FOLLOWER
I20260928 08:33:45.457772 30249 consensus_queue.cc:260] T 0fdf1988b86e4d81904fe880bd777beb P f996423f570146bdb87f10a69d734d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } } peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } }
I20260928 08:33:45.458962 30228 heartbeater.cc:503] Master 127.15.134.254:41481 was elected leader, sending a full tablet report...
I20260928 08:33:45.458755 30248 raft_consensus.cc:348] T 0fdf1988b86e4d81904fe880bd777beb P 28cdcfc4d9d14ccd91005a973261bfac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } } peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } }
I20260928 08:33:45.459276 30248 raft_consensus.cc:374] T 0fdf1988b86e4d81904fe880bd777beb P 28cdcfc4d9d14ccd91005a973261bfac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:33:45.459399 30248 raft_consensus.cc:729] T 0fdf1988b86e4d81904fe880bd777beb P 28cdcfc4d9d14ccd91005a973261bfac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28cdcfc4d9d14ccd91005a973261bfac, State: Initialized, Role: FOLLOWER
I20260928 08:33:45.460071 30248 consensus_queue.cc:260] T 0fdf1988b86e4d81904fe880bd777beb P 28cdcfc4d9d14ccd91005a973261bfac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } } peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } }
I20260928 08:33:45.461411 30249 ts_tablet_manager.cc:1428] T 0fdf1988b86e4d81904fe880bd777beb P f996423f570146bdb87f10a69d734d2c: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260928 08:33:45.463340 30248 ts_tablet_manager.cc:1428] T 0fdf1988b86e4d81904fe880bd777beb P 28cdcfc4d9d14ccd91005a973261bfac: Time spent starting tablet: real 0.016s user 0.013s sys 0.003s
I20260928 08:33:45.464326 30094 heartbeater.cc:503] Master 127.15.134.254:41481 was elected leader, sending a full tablet report...
I20260928 08:33:45.465610 30247 ts_tablet_manager.cc:1428] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b: Time spent starting tablet: real 0.020s user 0.019s sys 0.000s
W20260928 08:33:45.494096 29961 tablet.cc:2369] T 0fdf1988b86e4d81904fe880bd777beb P 28cdcfc4d9d14ccd91005a973261bfac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:33:45.533712 30095 tablet.cc:2369] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:33:45.577005 30229 tablet.cc:2369] T 0fdf1988b86e4d81904fe880bd777beb P f996423f570146bdb87f10a69d734d2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:33:45.671042 30253 raft_consensus.cc:482] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:33:45.671419 30253 raft_consensus.cc:504] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } } peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } }
I20260928 08:33:45.673326 30253 leader_election.cc:288] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28cdcfc4d9d14ccd91005a973261bfac (127.15.134.193:33821), f996423f570146bdb87f10a69d734d2c (127.15.134.195:44605)
I20260928 08:33:45.680673 30246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:45.673861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:43283 (local address 127.15.134.195:44605)
0928 08:33:45.674082 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:45.674088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:45.675029 (+ 941us) server_negotiation.cc:407] Connection header received
0928 08:33:45.675255 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:45.675261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:45.675337 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:45.675412 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:33:45.676033 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.676891 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.678875 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.679615 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.679810 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:45.679889 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:45.680220 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 08:33:45.680350 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 08:33:45.680668 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:45.673732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:57441 (local address 127.15.134.193:33821)
0928 08:33:45.673965 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:45.673972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:45.675096 (+ 1124us) server_negotiation.cc:407] Connection header received
0928 08:33:45.675300 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:45.675304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:45.675366 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:45.675476 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:33:45.676157 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.676976 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.678841 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.679616 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.679816 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:45.679900 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:45.680263 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 08:33:45.680389 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 08:33:45.680840 30256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:45.673768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:44605 (local address 127.15.134.194:43283)
0928 08:33:45.674892 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:45.674909 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:45.675089 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:45.675462 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:45.675466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:45.675491 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:33:45.675876 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:45.675885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.677025 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.677032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:45.678700 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:45.678712 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.679884 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.679890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:45.680053 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:45.680083 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 08:33:45.680159 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 08:33:45.680326 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"mutex_wait_us":55,"thread_start_us":658,"threads_started":1}
I20260928 08:33:45.680866 30257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:45.673857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:33821 (local address 127.15.134.194:57441)
0928 08:33:45.674947 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:45.674962 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:45.675128 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:45.675509 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:45.675513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:45.675538 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:33:45.676004 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:45.676016 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.677116 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.677120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:45.678681 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:45.678698 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.679855 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.679861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:45.680073 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:45.680096 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 08:33:45.680210 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 08:33:45.680341 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":992,"mutex_wait_us":83,"thread_start_us":662,"threads_started":1}
I20260928 08:33:45.681982 30183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fdf1988b86e4d81904fe880bd777beb" candidate_uuid: "84a23a8f68e047458b7932ff1803136b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f996423f570146bdb87f10a69d734d2c" is_pre_election: true
I20260928 08:33:45.682032 29915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fdf1988b86e4d81904fe880bd777beb" candidate_uuid: "84a23a8f68e047458b7932ff1803136b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28cdcfc4d9d14ccd91005a973261bfac" is_pre_election: true
I20260928 08:33:45.682526 30183 raft_consensus.cc:2454] T 0fdf1988b86e4d81904fe880bd777beb P f996423f570146bdb87f10a69d734d2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84a23a8f68e047458b7932ff1803136b in term 0.
I20260928 08:33:45.682525 29915 raft_consensus.cc:2454] T 0fdf1988b86e4d81904fe880bd777beb P 28cdcfc4d9d14ccd91005a973261bfac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84a23a8f68e047458b7932ff1803136b in term 0.
I20260928 08:33:45.683231 29982 leader_election.cc:302] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 28cdcfc4d9d14ccd91005a973261bfac, 84a23a8f68e047458b7932ff1803136b, f996423f570146bdb87f10a69d734d2c; no voters:
I20260928 08:33:45.683651 30253 raft_consensus.cc:2781] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:33:45.683836 30253 raft_consensus.cc:482] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:33:45.683974 30253 raft_consensus.cc:3037] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:33:45.686789 30253 raft_consensus.cc:504] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } } peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } }
I20260928 08:33:45.687453 30253 leader_election.cc:288] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b [CANDIDATE]: Term 1 election: Requested vote from peers 28cdcfc4d9d14ccd91005a973261bfac (127.15.134.193:33821), f996423f570146bdb87f10a69d734d2c (127.15.134.195:44605)
I20260928 08:33:45.688133 30183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fdf1988b86e4d81904fe880bd777beb" candidate_uuid: "84a23a8f68e047458b7932ff1803136b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f996423f570146bdb87f10a69d734d2c"
I20260928 08:33:45.688133 29915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fdf1988b86e4d81904fe880bd777beb" candidate_uuid: "84a23a8f68e047458b7932ff1803136b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28cdcfc4d9d14ccd91005a973261bfac"
I20260928 08:33:45.688390 29915 raft_consensus.cc:3037] T 0fdf1988b86e4d81904fe880bd777beb P 28cdcfc4d9d14ccd91005a973261bfac [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:33:45.688376 30183 raft_consensus.cc:3037] T 0fdf1988b86e4d81904fe880bd777beb P f996423f570146bdb87f10a69d734d2c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:33:45.691720 29915 raft_consensus.cc:2454] T 0fdf1988b86e4d81904fe880bd777beb P 28cdcfc4d9d14ccd91005a973261bfac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84a23a8f68e047458b7932ff1803136b in term 1.
I20260928 08:33:45.691720 30183 raft_consensus.cc:2454] T 0fdf1988b86e4d81904fe880bd777beb P f996423f570146bdb87f10a69d734d2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84a23a8f68e047458b7932ff1803136b in term 1.
I20260928 08:33:45.692214 29982 leader_election.cc:302] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84a23a8f68e047458b7932ff1803136b, f996423f570146bdb87f10a69d734d2c; no voters:
I20260928 08:33:45.692587 30253 raft_consensus.cc:2781] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:33:45.693192 30253 raft_consensus.cc:686] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b [term 1 LEADER]: Becoming Leader. State: Replica: 84a23a8f68e047458b7932ff1803136b, State: Running, Role: LEADER
I20260928 08:33:45.693719 30253 consensus_queue.cc:237] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } } peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } }
I20260928 08:33:45.699061 29770 catalog_manager.cc:5495] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b reported cstate change: term changed from 0 to 1, leader changed from <none> to 84a23a8f68e047458b7932ff1803136b (127.15.134.194). New cstate: current_term: 1 leader_uuid: "84a23a8f68e047458b7932ff1803136b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } health_report { overall_health: UNKNOWN } } }
I20260928 08:33:45.785483 30244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:45.778898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59896 (local address 127.15.134.194:46523)
0928 08:33:45.779087 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:45.779091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:45.779140 (+ 49us) server_negotiation.cc:407] Connection header received
0928 08:33:45.779425 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:45.779429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:45.779511 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:45.779659 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:33:45.780140 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.781026 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.783050 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.783486 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.783583 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:45.784136 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:33:45.784762 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:33:45.784928 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:45.785055 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 08:33:45.785138 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 08:33:45.795193 29915 raft_consensus.cc:1261] T 0fdf1988b86e4d81904fe880bd777beb P 28cdcfc4d9d14ccd91005a973261bfac [term 1 FOLLOWER]: Refusing update from remote peer 84a23a8f68e047458b7932ff1803136b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:33:45.795275 30183 raft_consensus.cc:1261] T 0fdf1988b86e4d81904fe880bd777beb P f996423f570146bdb87f10a69d734d2c [term 1 FOLLOWER]: Refusing update from remote peer 84a23a8f68e047458b7932ff1803136b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:33:45.796380 30258 consensus_queue.cc:1035] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b [LEADER]: Connected to new peer: Peer: permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:33:45.796797 30253 consensus_queue.cc:1035] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:33:45.808694 30262 mvcc.cc:205] Tried to move back new op lower bound from 7334233808045879296 to 7334233807645601792. Current Snapshot: MvccSnapshot[applied={T|T < 7334233808045879296}]
I20260928 08:33:45.824332 29771 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:33:45.824607 29771 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:33:45.828230 29771 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360: Sending DeleteTablet for 3 replicas of tablet 0fdf1988b86e4d81904fe880bd777beb
I20260928 08:33:45.829476 29895 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fdf1988b86e4d81904fe880bd777beb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:33:45 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:52996
I20260928 08:33:45.829624 30163 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fdf1988b86e4d81904fe880bd777beb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:33:45 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:53432
I20260928 08:33:45.829895 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 29832
I20260928 08:33:45.830614 30029 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fdf1988b86e4d81904fe880bd777beb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:33:45 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:59880
I20260928 08:33:45.843698 30270 tablet_replica.cc:330] T 0fdf1988b86e4d81904fe880bd777beb P 28cdcfc4d9d14ccd91005a973261bfac: stopping tablet replica
I20260928 08:33:45.844319 30270 raft_consensus.cc:2229] T 0fdf1988b86e4d81904fe880bd777beb P 28cdcfc4d9d14ccd91005a973261bfac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:33:45.844728 30273 tablet_replica.cc:330] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b: stopping tablet replica
I20260928 08:33:45.844941 30270 raft_consensus.cc:2258] T 0fdf1988b86e4d81904fe880bd777beb P 28cdcfc4d9d14ccd91005a973261bfac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:33:45.845307 30273 raft_consensus.cc:2229] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:33:45.847343 30270 ts_tablet_manager.cc:1905] T 0fdf1988b86e4d81904fe880bd777beb P 28cdcfc4d9d14ccd91005a973261bfac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:33:45.847970 30271 tablet_replica.cc:330] T 0fdf1988b86e4d81904fe880bd777beb P f996423f570146bdb87f10a69d734d2c: stopping tablet replica
I20260928 08:33:45.848347 30273 raft_consensus.cc:2258] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:33:45.848819 30271 raft_consensus.cc:2229] T 0fdf1988b86e4d81904fe880bd777beb P f996423f570146bdb87f10a69d734d2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:33:45.849331 30271 raft_consensus.cc:2258] T 0fdf1988b86e4d81904fe880bd777beb P f996423f570146bdb87f10a69d734d2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:33:45.851354 30271 ts_tablet_manager.cc:1905] T 0fdf1988b86e4d81904fe880bd777beb P f996423f570146bdb87f10a69d734d2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:33:45.851379 30273 ts_tablet_manager.cc:1905] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:33:45.854591 30270 ts_tablet_manager.cc:1918] T 0fdf1988b86e4d81904fe880bd777beb P 28cdcfc4d9d14ccd91005a973261bfac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:33:45.854844 30270 log.cc:1196] T 0fdf1988b86e4d81904fe880bd777beb P 28cdcfc4d9d14ccd91005a973261bfac: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/wal/wals/0fdf1988b86e4d81904fe880bd777beb
I20260928 08:33:45.855472 30270 ts_tablet_manager.cc:1939] T 0fdf1988b86e4d81904fe880bd777beb P 28cdcfc4d9d14ccd91005a973261bfac: Deleting consensus metadata
I20260928 08:33:45.857760 29757 catalog_manager.cc:4841] TS 28cdcfc4d9d14ccd91005a973261bfac (127.15.134.193:33821): tablet 0fdf1988b86e4d81904fe880bd777beb (table test-table [id=bd42f379d4c34fb19b715a55cd68de10]) successfully deleted
I20260928 08:33:45.858354 30271 ts_tablet_manager.cc:1918] T 0fdf1988b86e4d81904fe880bd777beb P f996423f570146bdb87f10a69d734d2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:33:45.858536 30271 log.cc:1196] T 0fdf1988b86e4d81904fe880bd777beb P f996423f570146bdb87f10a69d734d2c: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/0fdf1988b86e4d81904fe880bd777beb
I20260928 08:33:45.858995 30271 ts_tablet_manager.cc:1939] T 0fdf1988b86e4d81904fe880bd777beb P f996423f570146bdb87f10a69d734d2c: Deleting consensus metadata
I20260928 08:33:45.860817 29755 catalog_manager.cc:4841] TS f996423f570146bdb87f10a69d734d2c (127.15.134.195:44605): tablet 0fdf1988b86e4d81904fe880bd777beb (table test-table [id=bd42f379d4c34fb19b715a55cd68de10]) successfully deleted
I20260928 08:33:45.861472 30273 ts_tablet_manager.cc:1918] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:33:45.861660 30273 log.cc:1196] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/0fdf1988b86e4d81904fe880bd777beb
I20260928 08:33:45.862154 30273 ts_tablet_manager.cc:1939] T 0fdf1988b86e4d81904fe880bd777beb P 84a23a8f68e047458b7932ff1803136b: Deleting consensus metadata
I20260928 08:33:45.863656 29755 catalog_manager.cc:4841] TS 84a23a8f68e047458b7932ff1803136b (127.15.134.194:46523): tablet 0fdf1988b86e4d81904fe880bd777beb (table test-table [id=bd42f379d4c34fb19b715a55cd68de10]) successfully deleted
Sep 28 08:33:45 dist-test-slave-qp2n krb5kdc[29726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.193@KRBTEST.COM
I20260928 08:33:45.872184 30242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:45.831218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53004 (local address 127.15.134.193:33821)
0928 08:33:45.831455 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:45.831465 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:45.842823 (+ 11358us) server_negotiation.cc:407] Connection header received
0928 08:33:45.843102 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:45.843111 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:45.843237 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:45.843376 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:45.845697 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.847205 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.858769 (+ 11564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.859457 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.859571 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:45.865742 (+ 6171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:45.865758 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:45.865773 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:45.865811 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:45.869029 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:45.869882 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:45.869889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:45.869891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:45.870016 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:45.870477 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:45.870481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:45.870484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:45.870919 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:45.871182 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:45.871564 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 08:33:45.871743 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 08:33:45.873011 29955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:33:45.967216 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 29832
I20260928 08:33:45.991365 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 29966
Sep 28 08:33:45 dist-test-slave-qp2n krb5kdc[29726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.194@KRBTEST.COM
I20260928 08:33:46.010089 30244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:45.992223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59902 (local address 127.15.134.194:46523)
0928 08:33:45.992407 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:45.992412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:45.992476 (+ 64us) server_negotiation.cc:407] Connection header received
0928 08:33:45.992848 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:45.992853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:45.992923 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:45.993009 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:45.994856 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.995773 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.997811 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:45.998270 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:45.998386 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:46.003833 (+ 5447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:46.003848 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:46.003865 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:46.003908 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:46.007190 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:46.008005 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:46.008012 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:46.008015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:46.008179 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:46.008570 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:46.008574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:46.008576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:46.008995 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:46.009274 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:46.009566 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 08:33:46.009752 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 08:33:46.010856 30089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:33:46.127530 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 29966
I20260928 08:33:46.147936 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 30100
Sep 28 08:33:46 dist-test-slave-qp2n krb5kdc[29726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.195@KRBTEST.COM
I20260928 08:33:46.164271 30246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:46.148765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53434 (local address 127.15.134.195:44605)
0928 08:33:46.148962 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:46.148968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:46.149077 (+ 109us) server_negotiation.cc:407] Connection header received
0928 08:33:46.149302 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:46.149309 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:46.149384 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:46.149492 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:46.150817 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:46.151555 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:46.153124 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:46.153529 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:46.153669 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:46.158343 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:46.158357 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:46.158374 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:46.158414 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:46.161293 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:46.162056 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:46.162065 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:46.162067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:46.162222 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:46.162641 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:46.162650 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:46.162652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:46.163028 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:46.163355 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:46.163674 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 08:33:46.163904 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 08:33:46.165021 30223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:33:46.260895 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 30100
I20260928 08:33:46.279183 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 29739
I20260928 08:33:46.280380 29801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:33:46.393189 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 29739
I20260928 08:33:46.415346 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:41481
--webserver_interface=127.15.134.254
--webserver_port=42229
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:44545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:41481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:33:46.769939 30278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:33:46.770254 30278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:33:46.770344 30278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:33:46.778055 30278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:33:46.778215 30278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:33:46.778264 30278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:33:46.778314 30278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:33:46.778357 30278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:33:46.789145 30278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:44545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:41481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:41481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=42229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:33:46.790442 30278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:33:46.791850 30278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:33:46.817510 30287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:46.817519 30284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:46.817543 30285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:46.818444 30278 server_base.cc:1034] running on GCE node
I20260928 08:33:46.819320 30278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:33:46.821007 30278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:33:46.822292 30278 hybrid_clock.cc:648] HybridClock initialized: now 1790584426822254 us; error 68 us; skew 500 ppm
I20260928 08:33:46.822747 30278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:33:46 dist-test-slave-qp2n krb5kdc[29726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584426, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:33:46.826448 30278 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:33:46.828403 30278 webserver.cc:460] Webserver started at http://127.15.134.254:42229/ using document root <none> and password file <none>
I20260928 08:33:46.829255 30278 fs_manager.cc:359] Metadata directory not provided
I20260928 08:33:46.829427 30278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:33:46.835705 30278 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.001s
I20260928 08:33:46.838953 30294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:46.840481 30278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 08:33:46.840724 30278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "16349924272d46a988a8e12fa7eb8360"
format_stamp: "Formatted at 2026-09-28 08:33:43 on dist-test-slave-qp2n"
server_key: "3fa15451eb890da685b9562e5ec9303c"
server_key_iv: "56738fcd9f8f0b7460f5fb5a07e5e2d1"
server_key_version: "encryptionkey@0"
I20260928 08:33:46.841920 30278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:33:46.866866 30278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:33:46.871575 30278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:33:46.871893 30278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:33:46.892412 30278 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:41481
I20260928 08:33:46.892433 30345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:41481 every 8 connection(s)
I20260928 08:33:46.894099 30278 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:33:46.898906 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 30278
I20260928 08:33:46.899940 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:33821
--local_ip_for_outbound_sockets=127.15.134.193
--tserver_master_addrs=127.15.134.254:41481
--webserver_port=40457
--webserver_interface=127.15.134.193
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:44545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:33:46.903087 30346 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:33:46.908879 30346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360: Bootstrap starting.
I20260928 08:33:46.930120 30346 log.cc:824] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360: Log is configured to *not* fsync() on all Append() calls
I20260928 08:33:46.942914 30346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:33:46.943506 30346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360: Bootstrap complete.
I20260928 08:33:46.953444 30346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16349924272d46a988a8e12fa7eb8360" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41481 } }
I20260928 08:33:46.954300 30346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16349924272d46a988a8e12fa7eb8360, State: Initialized, Role: FOLLOWER
I20260928 08:33:46.955005 30346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16349924272d46a988a8e12fa7eb8360" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41481 } }
I20260928 08:33:46.955255 30346 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:33:46.955385 30346 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:33:46.955601 30346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [term 1 FOLLOWER]: Advancing to term 2
I20260928 08:33:46.959718 30346 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16349924272d46a988a8e12fa7eb8360" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41481 } }
I20260928 08:33:46.960255 30346 leader_election.cc:302] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16349924272d46a988a8e12fa7eb8360; no voters:
I20260928 08:33:46.960940 30346 leader_election.cc:288] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 08:33:46.961078 30350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 08:33:46.962251 30350 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [term 2 LEADER]: Becoming Leader. State: Replica: 16349924272d46a988a8e12fa7eb8360, State: Running, Role: LEADER
I20260928 08:33:46.962970 30350 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16349924272d46a988a8e12fa7eb8360" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41481 } }
I20260928 08:33:46.963560 30346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:33:46.965379 30352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16349924272d46a988a8e12fa7eb8360. Latest consensus state: current_term: 2 leader_uuid: "16349924272d46a988a8e12fa7eb8360" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16349924272d46a988a8e12fa7eb8360" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41481 } } }
I20260928 08:33:46.965775 30352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [sys.catalog]: This master's current role is: LEADER
I20260928 08:33:46.966208 30351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "16349924272d46a988a8e12fa7eb8360" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16349924272d46a988a8e12fa7eb8360" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41481 } } }
I20260928 08:33:46.966566 30351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [sys.catalog]: This master's current role is: LEADER
I20260928 08:33:46.975956 30356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:33:46.986194 30356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:33:46.988082 30356 catalog_manager.cc:1259] Loaded cluster ID: 0aaa7967c25b48dcbfdfa9ac3df6232e
I20260928 08:33:46.988266 30356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:33:46.992576 30356 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 08:33:47.022686 30356 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 08:33:47.038051 30356 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:33:47.040700 30356 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360: Loaded TSK: 0
I20260928 08:33:47.043154 30356 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 08:33:47.290912 30348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:33:47.291236 30348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:33:47.291399 30348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:33:47.299008 30348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:33:47.299170 30348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:33:47.299347 30348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:33:47.309702 30348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:44545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:33821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=40457
--tserver_master_addrs=127.15.134.254:41481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:33:47.311112 30348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:33:47.312484 30348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:33:47.328154 30376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:47.328161 30374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:47.328639 30373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:47.329564 30348 server_base.cc:1034] running on GCE node
I20260928 08:33:47.330200 30348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:33:47.331305 30348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:33:47.332536 30348 hybrid_clock.cc:648] HybridClock initialized: now 1790584427332501 us; error 35 us; skew 500 ppm
I20260928 08:33:47.333000 30348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:33:47 dist-test-slave-qp2n krb5kdc[29726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584427, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:33:47.336349 30348 init.cc:377] Logged in from keytab as kudu/127.15.134.193@KRBTEST.COM (short username kudu)
I20260928 08:33:47.338040 30348 webserver.cc:460] Webserver started at http://127.15.134.193:40457/ using document root <none> and password file <none>
I20260928 08:33:47.338860 30348 fs_manager.cc:359] Metadata directory not provided
I20260928 08:33:47.339007 30348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:33:47.344879 30348 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 08:33:47.348174 30383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:47.349871 30348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:33:47.350069 30348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "28cdcfc4d9d14ccd91005a973261bfac"
format_stamp: "Formatted at 2026-09-28 08:33:44 on dist-test-slave-qp2n"
server_key: "2ea04a7635602a12c19de0ca73dd8f59"
server_key_iv: "930d0735e018f10a56fb8bc660685917"
server_key_version: "encryptionkey@0"
I20260928 08:33:47.351102 30348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:33:47.374888 30348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:33:47.379925 30348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:33:47.380218 30348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:33:47.381311 30348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:33:47.382663 30348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:33:47.382802 30348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:47.382936 30348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:33:47.383025 30348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:47.416898 30348 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:33821
I20260928 08:33:47.416985 30495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:33821 every 8 connection(s)
I20260928 08:33:47.418457 30348 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:33:47.428377 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 30348
I20260928 08:33:47.429122 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:46523
--local_ip_for_outbound_sockets=127.15.134.194
--tserver_master_addrs=127.15.134.254:41481
--webserver_port=36563
--webserver_interface=127.15.134.194
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:44545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:33:47 dist-test-slave-qp2n krb5kdc[29726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584427, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:33:47.442840 30498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:47.421353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:52793 (local address 127.15.134.254:41481)
0928 08:33:47.421973 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:47.421984 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:47.422560 (+ 576us) server_negotiation.cc:407] Connection header received
0928 08:33:47.424588 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:47.424601 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:47.424984 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:47.425282 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:47.426628 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:47.427906 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:47.429934 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:47.431093 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:47.431339 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:47.434809 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:47.434820 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:47.434842 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:47.434891 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:47.438254 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:47.439224 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:47.439230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:47.439235 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:47.439409 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:47.439899 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:47.439903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:47.439905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:47.440379 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:47.440523 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:47.441516 (+ 993us) server_negotiation.cc:299] Negotiation successful
0928 08:33:47.441843 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":276,"threads_started":1}
I20260928 08:33:47.443279 30499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:47.421521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.193:52793)
0928 08:33:47.422355 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:47.422400 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:47.424118 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:47.425432 (+ 1314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:47.425439 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:47.425918 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:33:47.426438 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:47.426454 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:47.428075 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:47.428079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:47.429768 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:47.429780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:47.431364 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:47.431370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:47.431553 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:47.432242 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:33:47.432269 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:33:47.434512 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:33:47.438437 (+ 3925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:47.438449 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:47.438467 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:47.439095 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:47.439522 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:47.439530 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:47.439536 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:47.439788 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:47.440520 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:33:47.440531 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:33:47.441023 (+ 492us) client_negotiation.cc:769] Sending connection context
0928 08:33:47.441334 (+ 311us) client_negotiation.cc:240] Negotiation successful
0928 08:33:47.441927 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":355,"threads_started":1}
I20260928 08:33:47.446182 30496 heartbeater.cc:347] Connected to a master server at 127.15.134.254:41481
I20260928 08:33:47.446522 30496 heartbeater.cc:464] Registering TS with master...
I20260928 08:33:47.447444 30496 heartbeater.cc:511] Master 127.15.134.254:41481 requested a full tablet report, sending...
I20260928 08:33:47.450142 30311 ts_manager.cc:194] Registered new tserver with Master: 28cdcfc4d9d14ccd91005a973261bfac (127.15.134.193:33821)
I20260928 08:33:47.452543 30311 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.193@KRBTEST.COM'} at 127.15.134.193:52793
W20260928 08:33:47.778323 30500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:33:47.778602 30500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:33:47.778698 30500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:33:47.785791 30500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:33:47.785897 30500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:33:47.786010 30500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:33:47.795512 30500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:44545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:46523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=36563
--tserver_master_addrs=127.15.134.254:41481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:33:47.796784 30500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:33:47.798023 30500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:33:47.813441 30505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:47.813400 30506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:47.813894 30508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:47.814623 30500 server_base.cc:1034] running on GCE node
I20260928 08:33:47.815199 30500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:33:47.816192 30500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:33:47.817456 30500 hybrid_clock.cc:648] HybridClock initialized: now 1790584427817413 us; error 35 us; skew 500 ppm
I20260928 08:33:47.817828 30500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:33:47 dist-test-slave-qp2n krb5kdc[29726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584427, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:33:47.821280 30500 init.cc:377] Logged in from keytab as kudu/127.15.134.194@KRBTEST.COM (short username kudu)
I20260928 08:33:47.822964 30500 webserver.cc:460] Webserver started at http://127.15.134.194:36563/ using document root <none> and password file <none>
I20260928 08:33:47.823701 30500 fs_manager.cc:359] Metadata directory not provided
I20260928 08:33:47.823863 30500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:33:47.829752 30500 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 08:33:47.832777 30515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:47.834311 30500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:33:47.834538 30500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "84a23a8f68e047458b7932ff1803136b"
format_stamp: "Formatted at 2026-09-28 08:33:44 on dist-test-slave-qp2n"
server_key: "788e99bc59a6ee4d7bec42261b4a3fb2"
server_key_iv: "b3cc99c786929eccc3217f1290c8d219"
server_key_version: "encryptionkey@0"
I20260928 08:33:47.835596 30500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:33:47.871533 30500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:33:47.876219 30500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:33:47.876526 30500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:33:47.877575 30500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:33:47.878963 30500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:33:47.879056 30500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:47.879140 30500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:33:47.879196 30500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:47.913990 30500 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:46523
I20260928 08:33:47.914026 30627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:46523 every 8 connection(s)
I20260928 08:33:47.915398 30500 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:33:47.917089 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 30500
I20260928 08:33:47.917800 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:44605
--local_ip_for_outbound_sockets=127.15.134.195
--tserver_master_addrs=127.15.134.254:41481
--webserver_port=34823
--webserver_interface=127.15.134.195
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:44545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:33:47 dist-test-slave-qp2n krb5kdc[29726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584427, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:33:47.939783 30498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:47.918147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:43837 (local address 127.15.134.254:41481)
0928 08:33:47.918379 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:47.918386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:47.919731 (+ 1345us) server_negotiation.cc:407] Connection header received
0928 08:33:47.922648 (+ 2917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:47.922653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:47.922762 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:47.922864 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:47.924941 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:47.925827 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:47.927390 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:47.928248 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:47.928408 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:47.931993 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:47.932004 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:47.932012 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:47.932074 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:47.935976 (+ 3902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:47.936810 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:47.936820 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:47.936827 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:47.937015 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:47.937565 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:47.937586 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:47.937591 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:47.938056 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:47.938251 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:47.938937 (+ 686us) server_negotiation.cc:299] Negotiation successful
0928 08:33:47.939234 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 08:33:47.940536 30630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:47.918737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.194:43837)
0928 08:33:47.919554 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:47.919604 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:47.922330 (+ 2726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:47.923142 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:47.923157 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:47.923909 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:33:47.924730 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:47.924758 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:47.926013 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:47.926017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:47.927218 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:47.927230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:47.928457 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:47.928461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:47.928726 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:47.929438 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:33:47.929459 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:33:47.931748 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:33:47.936138 (+ 4390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:47.936146 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:47.936160 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:47.936666 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:47.937136 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:47.937139 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:47.937141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:47.937425 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:47.938187 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:33:47.938193 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:33:47.938535 (+ 342us) client_negotiation.cc:769] Sending connection context
0928 08:33:47.938762 (+ 227us) client_negotiation.cc:240] Negotiation successful
0928 08:33:47.939237 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":298,"threads_started":1}
I20260928 08:33:47.942153 30628 heartbeater.cc:347] Connected to a master server at 127.15.134.254:41481
I20260928 08:33:47.942497 30628 heartbeater.cc:464] Registering TS with master...
I20260928 08:33:47.943292 30628 heartbeater.cc:511] Master 127.15.134.254:41481 requested a full tablet report, sending...
I20260928 08:33:47.944999 30311 ts_manager.cc:194] Registered new tserver with Master: 84a23a8f68e047458b7932ff1803136b (127.15.134.194:46523)
I20260928 08:33:47.946013 30311 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.194@KRBTEST.COM'} at 127.15.134.194:43837
W20260928 08:33:48.262949 30631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:33:48.263346 30631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:33:48.263535 30631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:33:48.271472 30631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:33:48.271646 30631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:33:48.271832 30631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:33:48.282761 30631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:44545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:44605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=34823
--tserver_master_addrs=127.15.134.254:41481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:33:48.284253 30631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:33:48.285683 30631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:33:48.299001 30637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:48.299289 30636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:48.299026 30639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:48.299659 30631 server_base.cc:1034] running on GCE node
I20260928 08:33:48.300192 30631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:33:48.301182 30631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:33:48.302413 30631 hybrid_clock.cc:648] HybridClock initialized: now 1790584428302336 us; error 57 us; skew 500 ppm
I20260928 08:33:48.302838 30631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:33:48 dist-test-slave-qp2n krb5kdc[29726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584428, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:33:48.306053 30631 init.cc:377] Logged in from keytab as kudu/127.15.134.195@KRBTEST.COM (short username kudu)
I20260928 08:33:48.307554 30631 webserver.cc:460] Webserver started at http://127.15.134.195:34823/ using document root <none> and password file <none>
I20260928 08:33:48.308252 30631 fs_manager.cc:359] Metadata directory not provided
I20260928 08:33:48.308411 30631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:33:48.313868 30631 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:33:48.316649 30646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:48.318202 30631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:33:48.318390 30631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "f996423f570146bdb87f10a69d734d2c"
format_stamp: "Formatted at 2026-09-28 08:33:45 on dist-test-slave-qp2n"
server_key: "397a852c371f5c109bda49e7d3b9ef39"
server_key_iv: "74a5e53a3c35c6f34c1b47333d20504e"
server_key_version: "encryptionkey@0"
I20260928 08:33:48.319346 30631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:33:48.337729 30631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:33:48.342595 30631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:33:48.342901 30631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:33:48.343971 30631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:33:48.345582 30631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:33:48.345738 30631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:48.345888 30631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:33:48.345979 30631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:48.382607 30631 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:44605
I20260928 08:33:48.382647 30758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:44605 every 8 connection(s)
I20260928 08:33:48.384328 30631 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:33:48.385829 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 30631
Sep 28 08:33:48 dist-test-slave-qp2n krb5kdc[29726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584428, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:33:48.404110 30498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:48.387158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49270 (local address 127.15.134.254:41481)
0928 08:33:48.387382 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:48.387391 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:48.388078 (+ 687us) server_negotiation.cc:407] Connection header received
0928 08:33:48.388303 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:48.388309 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:48.388428 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:48.388649 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:48.390921 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.391764 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.394044 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.394569 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.394865 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:48.397696 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:48.397701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:48.397704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:48.397737 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:48.400898 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:48.401621 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:48.401627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:48.401631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:48.401775 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:48.402213 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:48.402218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:48.402221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:48.402630 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:48.402874 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:48.403303 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 08:33:48.403525 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":80}
I20260928 08:33:48.409201 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:48.387291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:47465 (local address 127.15.134.254:41481)
0928 08:33:48.391040 (+ 3749us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:48.391049 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:48.391084 (+ 35us) server_negotiation.cc:407] Connection header received
0928 08:33:48.392140 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:48.392144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:48.392206 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:48.392295 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:48.394447 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.395367 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.397130 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.397631 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.397784 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:48.401796 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:48.401801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:48.401805 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:48.401838 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:48.405629 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:48.406711 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:48.406716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:48.406718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:48.406912 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:48.407428 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:48.407431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:48.407434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:48.407822 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:48.407950 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:48.408743 (+ 793us) server_negotiation.cc:299] Negotiation successful
0928 08:33:48.408921 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3591,"thread_start_us":1129,"threads_started":1}
I20260928 08:33:48.410147 30763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:48.387977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.195:47465)
0928 08:33:48.389138 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:48.389203 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:48.391784 (+ 2581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:48.392588 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:48.392600 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:48.393400 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:33:48.394216 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:48.394237 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.395559 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.395564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:48.396963 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:48.396976 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.397874 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.397884 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:48.398172 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:48.398906 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:33:48.398930 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:33:48.401494 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:33:48.405778 (+ 4284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:48.405791 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:48.405811 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:48.406482 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:48.407045 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:48.407050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:48.407054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:48.407304 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:48.407942 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:33:48.407948 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:33:48.408300 (+ 352us) client_negotiation.cc:769] Sending connection context
0928 08:33:48.408540 (+ 240us) client_negotiation.cc:240] Negotiation successful
0928 08:33:48.409013 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"thread_start_us":543,"threads_started":1}
I20260928 08:33:48.411494 30759 heartbeater.cc:347] Connected to a master server at 127.15.134.254:41481
I20260928 08:33:48.411809 30759 heartbeater.cc:464] Registering TS with master...
I20260928 08:33:48.412550 30759 heartbeater.cc:511] Master 127.15.134.254:41481 requested a full tablet report, sending...
I20260928 08:33:48.414047 30311 ts_manager.cc:194] Registered new tserver with Master: f996423f570146bdb87f10a69d734d2c (127.15.134.195:44605)
I20260928 08:33:48.414913 30311 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.195@KRBTEST.COM'} at 127.15.134.195:47465
I20260928 08:33:48.424917 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:33:48.432245 30762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:48.426051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49272 (local address 127.15.134.254:41481)
0928 08:33:48.426266 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:48.426271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:48.426697 (+ 426us) server_negotiation.cc:407] Connection header received
0928 08:33:48.426881 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:48.426885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:48.427026 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:48.427180 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:33:48.427579 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.428352 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.429815 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.430302 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.430405 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:48.430910 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:33:48.431662 (+ 752us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:33:48.431788 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:48.431932 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 08:33:48.432007 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 08:33:48.433437 30311 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:33:48.441385 30311 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:33:48.456364 30496 heartbeater.cc:503] Master 127.15.134.254:41481 was elected leader, sending a full tablet report...
I20260928 08:33:48.477047 30769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:48.455449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:46523 (local address 127.0.0.1:50438)
0928 08:33:48.457283 (+ 1834us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:48.457348 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:48.457563 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:48.469032 (+ 11469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:48.469044 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:48.469115 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:33:48.469819 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:48.469834 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.471668 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.471676 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:48.474096 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:48.474109 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.475899 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.475906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:48.476112 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:48.476168 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 08:33:48.476294 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 08:33:48.476520 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1466,"mutex_wait_us":105,"spinlock_wait_cycles":14720,"thread_start_us":1447,"threads_started":1}
I20260928 08:33:48.477521 30773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:48.461587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:44605 (local address 127.0.0.1:60982)
0928 08:33:48.469561 (+ 7974us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:48.469583 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:48.469744 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:48.470388 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:48.470392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:48.470419 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:33:48.470808 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:48.470820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.472784 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.472793 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:48.474919 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:48.474930 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.476725 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.476732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:48.476912 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:48.476944 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 08:33:48.477053 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 08:33:48.477204 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7847,"thread_start_us":7677,"threads_started":1}
I20260928 08:33:48.477787 30771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:48.455656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50438 (local address 127.15.134.194:46523)
0928 08:33:48.467269 (+ 11613us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:48.468104 (+ 835us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:48.468141 (+ 37us) server_negotiation.cc:407] Connection header received
0928 08:33:48.468300 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:48.468312 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:48.468697 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:48.469907 (+ 1210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:33:48.470139 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.471534 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.474262 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.475668 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.475847 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:48.476033 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:48.476764 (+ 731us) server_negotiation.cc:299] Negotiation successful
0928 08:33:48.476953 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11434,"thread_start_us":12188,"threads_started":1}
I20260928 08:33:48.477962 30774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:48.462292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60982 (local address 127.15.134.195:44605)
0928 08:33:48.468745 (+ 6453us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:48.468751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:48.469890 (+ 1139us) server_negotiation.cc:407] Connection header received
0928 08:33:48.470000 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:48.470008 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:48.470169 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:48.470381 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:33:48.471009 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.472576 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.475077 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.476152 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.476359 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:48.476515 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:48.477346 (+ 831us) server_negotiation.cc:299] Negotiation successful
0928 08:33:48.477518 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6268,"thread_start_us":6100,"threads_started":1}
I20260928 08:33:48.478531 30770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:48.455593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:33821 (local address 127.0.0.1:47084)
0928 08:33:48.466667 (+ 11074us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:48.466693 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:48.468863 (+ 2170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:48.469633 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:48.469637 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:48.469665 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:33:48.470114 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:48.470125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.472022 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.472028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:48.474547 (+ 2519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:48.474564 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.476411 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.476417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:48.476661 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:48.476709 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 08:33:48.476800 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 08:33:48.476933 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10891,"mutex_wait_us":2,"thread_start_us":10727,"threads_started":1}
I20260928 08:33:48.479127 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:48.456265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47084 (local address 127.15.134.193:33821)
0928 08:33:48.461866 (+ 5601us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:48.461879 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:48.466832 (+ 4953us) server_negotiation.cc:407] Connection header received
0928 08:33:48.469133 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:48.469145 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:48.469434 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:48.469648 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:33:48.470310 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.471841 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.474726 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.476160 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.476377 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:48.476550 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:48.477456 (+ 906us) server_negotiation.cc:299] Negotiation successful
0928 08:33:48.477830 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5396,"thread_start_us":5245,"threads_started":1}
I20260928 08:33:48.481609 30694 tablet_service.cc:1467] Processing CreateTablet for tablet 2129daeb3c394d96a0a94e36bfa2d21d (DEFAULT_TABLE table=test-table [id=ca962e0a28e447d2bd1cd2b9a79a89f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:33:48.481740 30563 tablet_service.cc:1467] Processing CreateTablet for tablet 2129daeb3c394d96a0a94e36bfa2d21d (DEFAULT_TABLE table=test-table [id=ca962e0a28e447d2bd1cd2b9a79a89f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:33:48.482892 30694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2129daeb3c394d96a0a94e36bfa2d21d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:33:48.483055 30563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2129daeb3c394d96a0a94e36bfa2d21d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:33:48.482657 30431 tablet_service.cc:1467] Processing CreateTablet for tablet 2129daeb3c394d96a0a94e36bfa2d21d (DEFAULT_TABLE table=test-table [id=ca962e0a28e447d2bd1cd2b9a79a89f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:33:48.483958 30431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2129daeb3c394d96a0a94e36bfa2d21d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:33:48.496099 30775 tablet_bootstrap.cc:492] T 2129daeb3c394d96a0a94e36bfa2d21d P f996423f570146bdb87f10a69d734d2c: Bootstrap starting.
I20260928 08:33:48.498453 30776 tablet_bootstrap.cc:492] T 2129daeb3c394d96a0a94e36bfa2d21d P 84a23a8f68e047458b7932ff1803136b: Bootstrap starting.
I20260928 08:33:48.500109 30775 tablet_bootstrap.cc:654] T 2129daeb3c394d96a0a94e36bfa2d21d P f996423f570146bdb87f10a69d734d2c: Neither blocks nor log segments found. Creating new log.
I20260928 08:33:48.500460 30777 tablet_bootstrap.cc:492] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac: Bootstrap starting.
I20260928 08:33:48.501621 30775 log.cc:824] T 2129daeb3c394d96a0a94e36bfa2d21d P f996423f570146bdb87f10a69d734d2c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:33:48.502002 30776 tablet_bootstrap.cc:654] T 2129daeb3c394d96a0a94e36bfa2d21d P 84a23a8f68e047458b7932ff1803136b: Neither blocks nor log segments found. Creating new log.
I20260928 08:33:48.503126 30776 log.cc:824] T 2129daeb3c394d96a0a94e36bfa2d21d P 84a23a8f68e047458b7932ff1803136b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:33:48.503566 30777 tablet_bootstrap.cc:654] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac: Neither blocks nor log segments found. Creating new log.
I20260928 08:33:48.504803 30775 tablet_bootstrap.cc:492] T 2129daeb3c394d96a0a94e36bfa2d21d P f996423f570146bdb87f10a69d734d2c: No bootstrap required, opened a new log
I20260928 08:33:48.504812 30777 log.cc:824] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac: Log is configured to *not* fsync() on all Append() calls
I20260928 08:33:48.505070 30775 ts_tablet_manager.cc:1397] T 2129daeb3c394d96a0a94e36bfa2d21d P f996423f570146bdb87f10a69d734d2c: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260928 08:33:48.506268 30776 tablet_bootstrap.cc:492] T 2129daeb3c394d96a0a94e36bfa2d21d P 84a23a8f68e047458b7932ff1803136b: No bootstrap required, opened a new log
I20260928 08:33:48.506532 30776 ts_tablet_manager.cc:1397] T 2129daeb3c394d96a0a94e36bfa2d21d P 84a23a8f68e047458b7932ff1803136b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 08:33:48.507910 30777 tablet_bootstrap.cc:492] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac: No bootstrap required, opened a new log
I20260928 08:33:48.508157 30777 ts_tablet_manager.cc:1397] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260928 08:33:48.514910 30777 raft_consensus.cc:348] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } } peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } }
I20260928 08:33:48.515297 30777 raft_consensus.cc:374] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:33:48.515388 30777 raft_consensus.cc:729] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28cdcfc4d9d14ccd91005a973261bfac, State: Initialized, Role: FOLLOWER
I20260928 08:33:48.516009 30777 consensus_queue.cc:260] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } } peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } }
I20260928 08:33:48.515888 30775 raft_consensus.cc:348] T 2129daeb3c394d96a0a94e36bfa2d21d P f996423f570146bdb87f10a69d734d2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } } peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } }
I20260928 08:33:48.516274 30775 raft_consensus.cc:374] T 2129daeb3c394d96a0a94e36bfa2d21d P f996423f570146bdb87f10a69d734d2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:33:48.516371 30775 raft_consensus.cc:729] T 2129daeb3c394d96a0a94e36bfa2d21d P f996423f570146bdb87f10a69d734d2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f996423f570146bdb87f10a69d734d2c, State: Initialized, Role: FOLLOWER
I20260928 08:33:48.517076 30775 consensus_queue.cc:260] T 2129daeb3c394d96a0a94e36bfa2d21d P f996423f570146bdb87f10a69d734d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } } peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } }
I20260928 08:33:48.516997 30776 raft_consensus.cc:348] T 2129daeb3c394d96a0a94e36bfa2d21d P 84a23a8f68e047458b7932ff1803136b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } } peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } }
I20260928 08:33:48.517431 30776 raft_consensus.cc:374] T 2129daeb3c394d96a0a94e36bfa2d21d P 84a23a8f68e047458b7932ff1803136b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:33:48.517536 30776 raft_consensus.cc:729] T 2129daeb3c394d96a0a94e36bfa2d21d P 84a23a8f68e047458b7932ff1803136b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84a23a8f68e047458b7932ff1803136b, State: Initialized, Role: FOLLOWER
I20260928 08:33:48.517963 30777 ts_tablet_manager.cc:1428] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac: Time spent starting tablet: real 0.010s user 0.003s sys 0.007s
I20260928 08:33:48.518385 30776 consensus_queue.cc:260] T 2129daeb3c394d96a0a94e36bfa2d21d P 84a23a8f68e047458b7932ff1803136b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } } peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } }
I20260928 08:33:48.520944 30759 heartbeater.cc:503] Master 127.15.134.254:41481 was elected leader, sending a full tablet report...
I20260928 08:33:48.521733 30628 heartbeater.cc:503] Master 127.15.134.254:41481 was elected leader, sending a full tablet report...
I20260928 08:33:48.521775 30775 ts_tablet_manager.cc:1428] T 2129daeb3c394d96a0a94e36bfa2d21d P f996423f570146bdb87f10a69d734d2c: Time spent starting tablet: real 0.017s user 0.014s sys 0.001s
I20260928 08:33:48.523324 30776 ts_tablet_manager.cc:1428] T 2129daeb3c394d96a0a94e36bfa2d21d P 84a23a8f68e047458b7932ff1803136b: Time spent starting tablet: real 0.017s user 0.010s sys 0.007s
I20260928 08:33:48.533043 30781 raft_consensus.cc:482] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:33:48.533375 30781 raft_consensus.cc:504] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } } peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } }
I20260928 08:33:48.535089 30781 leader_election.cc:288] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84a23a8f68e047458b7932ff1803136b (127.15.134.194:46523), f996423f570146bdb87f10a69d734d2c (127.15.134.195:44605)
I20260928 08:33:48.543839 30774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:48.536372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:41141 (local address 127.15.134.195:44605)
0928 08:33:48.536576 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:48.536582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:48.536912 (+ 330us) server_negotiation.cc:407] Connection header received
0928 08:33:48.537168 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:48.537172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:48.537240 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:48.537332 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:33:48.537839 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.538885 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.541526 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.542775 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.542948 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:48.543035 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:48.543461 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 08:33:48.543590 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 08:33:48.543921 30771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:48.535390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:49453 (local address 127.15.134.194:46523)
0928 08:33:48.535650 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:48.535660 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:48.536189 (+ 529us) server_negotiation.cc:407] Connection header received
0928 08:33:48.536521 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:48.536529 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:48.536640 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:48.536728 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:33:48.537488 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.538685 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.541526 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.542908 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.543057 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:48.543129 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:48.543471 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 08:33:48.543689 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 08:33:48.544049 30784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:48.535312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:46523 (local address 127.15.134.193:49453)
0928 08:33:48.536087 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:48.536108 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:48.536307 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:48.536939 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:48.536943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:48.536971 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:33:48.537293 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:48.537327 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.538842 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.538846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:48.541342 (+ 2496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:48.541359 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.543107 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.543114 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:48.543321 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:48.543354 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 08:33:48.543431 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 08:33:48.543546 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"mutex_wait_us":50,"thread_start_us":393,"threads_started":1}
I20260928 08:33:48.544076 30785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:48.536173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:44605 (local address 127.15.134.193:41141)
0928 08:33:48.536819 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:48.536835 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:48.536944 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:48.537369 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:48.537376 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:48.537405 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:33:48.537679 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:48.537688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.539035 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.539040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:48.541351 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:48.541359 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.543019 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.543028 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:48.543263 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:48.543303 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 08:33:48.543405 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 08:33:48.543546 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":67,"thread_start_us":296,"threads_started":1}
I20260928 08:33:48.545254 30583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2129daeb3c394d96a0a94e36bfa2d21d" candidate_uuid: "28cdcfc4d9d14ccd91005a973261bfac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84a23a8f68e047458b7932ff1803136b" is_pre_election: true
I20260928 08:33:48.545332 30714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2129daeb3c394d96a0a94e36bfa2d21d" candidate_uuid: "28cdcfc4d9d14ccd91005a973261bfac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f996423f570146bdb87f10a69d734d2c" is_pre_election: true
I20260928 08:33:48.545796 30583 raft_consensus.cc:2454] T 2129daeb3c394d96a0a94e36bfa2d21d P 84a23a8f68e047458b7932ff1803136b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28cdcfc4d9d14ccd91005a973261bfac in term 0.
I20260928 08:33:48.545970 30714 raft_consensus.cc:2454] T 2129daeb3c394d96a0a94e36bfa2d21d P f996423f570146bdb87f10a69d734d2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28cdcfc4d9d14ccd91005a973261bfac in term 0.
I20260928 08:33:48.546540 30384 leader_election.cc:302] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28cdcfc4d9d14ccd91005a973261bfac, 84a23a8f68e047458b7932ff1803136b; no voters:
I20260928 08:33:48.546959 30781 raft_consensus.cc:2781] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:33:48.547119 30781 raft_consensus.cc:482] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:33:48.547224 30781 raft_consensus.cc:3037] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:33:48.549474 30781 raft_consensus.cc:504] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } } peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } }
I20260928 08:33:48.550077 30781 leader_election.cc:288] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [CANDIDATE]: Term 1 election: Requested vote from peers 84a23a8f68e047458b7932ff1803136b (127.15.134.194:46523), f996423f570146bdb87f10a69d734d2c (127.15.134.195:44605)
I20260928 08:33:48.550525 30583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2129daeb3c394d96a0a94e36bfa2d21d" candidate_uuid: "28cdcfc4d9d14ccd91005a973261bfac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84a23a8f68e047458b7932ff1803136b"
I20260928 08:33:48.550596 30714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2129daeb3c394d96a0a94e36bfa2d21d" candidate_uuid: "28cdcfc4d9d14ccd91005a973261bfac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f996423f570146bdb87f10a69d734d2c"
I20260928 08:33:48.550714 30583 raft_consensus.cc:3037] T 2129daeb3c394d96a0a94e36bfa2d21d P 84a23a8f68e047458b7932ff1803136b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:33:48.550900 30714 raft_consensus.cc:3037] T 2129daeb3c394d96a0a94e36bfa2d21d P f996423f570146bdb87f10a69d734d2c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:33:48.553097 30583 raft_consensus.cc:2454] T 2129daeb3c394d96a0a94e36bfa2d21d P 84a23a8f68e047458b7932ff1803136b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28cdcfc4d9d14ccd91005a973261bfac in term 1.
I20260928 08:33:48.553253 30714 raft_consensus.cc:2454] T 2129daeb3c394d96a0a94e36bfa2d21d P f996423f570146bdb87f10a69d734d2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28cdcfc4d9d14ccd91005a973261bfac in term 1.
I20260928 08:33:48.553601 30384 leader_election.cc:302] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28cdcfc4d9d14ccd91005a973261bfac, 84a23a8f68e047458b7932ff1803136b; no voters:
I20260928 08:33:48.553965 30781 raft_consensus.cc:2781] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:33:48.554495 30781 raft_consensus.cc:686] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 1 LEADER]: Becoming Leader. State: Replica: 28cdcfc4d9d14ccd91005a973261bfac, State: Running, Role: LEADER
I20260928 08:33:48.555186 30781 consensus_queue.cc:237] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } } peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } }
I20260928 08:33:48.560144 30311 catalog_manager.cc:5495] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac reported cstate change: term changed from 0 to 1, leader changed from <none> to 28cdcfc4d9d14ccd91005a973261bfac (127.15.134.193). New cstate: current_term: 1 leader_uuid: "28cdcfc4d9d14ccd91005a973261bfac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } health_report { overall_health: UNKNOWN } } }
I20260928 08:33:48.593387 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:48.587933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47100 (local address 127.15.134.193:33821)
0928 08:33:48.588140 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:48.588145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:48.588162 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:33:48.588279 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:48.588283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:48.588358 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:48.588484 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:33:48.589001 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.589742 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.591275 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.591673 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.591771 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:48.592198 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:33:48.592786 (+ 588us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:33:48.592895 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:48.593001 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 08:33:48.593064 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 08:33:48.601333 30583 raft_consensus.cc:1261] T 2129daeb3c394d96a0a94e36bfa2d21d P 84a23a8f68e047458b7932ff1803136b [term 1 FOLLOWER]: Refusing update from remote peer 28cdcfc4d9d14ccd91005a973261bfac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:33:48.601390 30714 raft_consensus.cc:1261] T 2129daeb3c394d96a0a94e36bfa2d21d P f996423f570146bdb87f10a69d734d2c [term 1 FOLLOWER]: Refusing update from remote peer 28cdcfc4d9d14ccd91005a973261bfac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:33:48.602310 30781 consensus_queue.cc:1035] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [LEADER]: Connected to new peer: Peer: permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:33:48.602630 30786 consensus_queue.cc:1035] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [LEADER]: Connected to new peer: Peer: permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:33:48.629312 30311 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:33:48.629575 30311 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:33:48.633445 30311 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360: Sending DeleteTablet for 3 replicas of tablet 2129daeb3c394d96a0a94e36bfa2d21d
I20260928 08:33:48.634756 30431 tablet_service.cc:1514] Processing DeleteTablet for tablet 2129daeb3c394d96a0a94e36bfa2d21d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:33:48 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:47084
I20260928 08:33:48.634938 30563 tablet_service.cc:1514] Processing DeleteTablet for tablet 2129daeb3c394d96a0a94e36bfa2d21d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:33:48 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:50438
I20260928 08:33:48.635159 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 30348
I20260928 08:33:48.635921 30694 tablet_service.cc:1514] Processing DeleteTablet for tablet 2129daeb3c394d96a0a94e36bfa2d21d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:33:48 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:60982
I20260928 08:33:48.660360 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:48.637000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47110 (local address 127.15.134.193:33821)
0928 08:33:48.637173 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:48.637179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:48.637199 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:33:48.637255 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:48.637259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:48.637336 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:48.637448 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:48.639785 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.641293 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.643699 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.644690 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.644831 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:48.649713 (+ 4882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:48.649732 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:48.649756 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:48.649814 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:48.654945 (+ 5131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:48.656207 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:48.656220 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:48.656227 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:48.656437 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:48.657006 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:48.657014 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:48.657019 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:48.657617 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:48.657953 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:48.659486 (+ 1533us) server_negotiation.cc:299] Negotiation successful
0928 08:33:48.659803 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 08:33:48.661535 30491 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:33:48.661715 30801 tablet_replica.cc:330] T 2129daeb3c394d96a0a94e36bfa2d21d P f996423f570146bdb87f10a69d734d2c: stopping tablet replica
I20260928 08:33:48.662303 30801 raft_consensus.cc:2229] T 2129daeb3c394d96a0a94e36bfa2d21d P f996423f570146bdb87f10a69d734d2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:33:48.663139 30801 raft_consensus.cc:2258] T 2129daeb3c394d96a0a94e36bfa2d21d P f996423f570146bdb87f10a69d734d2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:33:48.663971 30800 tablet_replica.cc:330] T 2129daeb3c394d96a0a94e36bfa2d21d P 84a23a8f68e047458b7932ff1803136b: stopping tablet replica
I20260928 08:33:48.664546 30800 raft_consensus.cc:2229] T 2129daeb3c394d96a0a94e36bfa2d21d P 84a23a8f68e047458b7932ff1803136b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:33:48.665285 30800 raft_consensus.cc:2258] T 2129daeb3c394d96a0a94e36bfa2d21d P 84a23a8f68e047458b7932ff1803136b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:33:48.665763 30801 ts_tablet_manager.cc:1905] T 2129daeb3c394d96a0a94e36bfa2d21d P f996423f570146bdb87f10a69d734d2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:33:48.668404 30800 ts_tablet_manager.cc:1905] T 2129daeb3c394d96a0a94e36bfa2d21d P 84a23a8f68e047458b7932ff1803136b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:33:48.677274 30801 ts_tablet_manager.cc:1918] T 2129daeb3c394d96a0a94e36bfa2d21d P f996423f570146bdb87f10a69d734d2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:33:48.677532 30801 log.cc:1196] T 2129daeb3c394d96a0a94e36bfa2d21d P f996423f570146bdb87f10a69d734d2c: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/2129daeb3c394d96a0a94e36bfa2d21d
I20260928 08:33:48.678241 30801 ts_tablet_manager.cc:1939] T 2129daeb3c394d96a0a94e36bfa2d21d P f996423f570146bdb87f10a69d734d2c: Deleting consensus metadata
I20260928 08:33:48.678459 30800 ts_tablet_manager.cc:1918] T 2129daeb3c394d96a0a94e36bfa2d21d P 84a23a8f68e047458b7932ff1803136b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:33:48.678643 30800 log.cc:1196] T 2129daeb3c394d96a0a94e36bfa2d21d P 84a23a8f68e047458b7932ff1803136b: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/2129daeb3c394d96a0a94e36bfa2d21d
I20260928 08:33:48.679230 30800 ts_tablet_manager.cc:1939] T 2129daeb3c394d96a0a94e36bfa2d21d P 84a23a8f68e047458b7932ff1803136b: Deleting consensus metadata
I20260928 08:33:48.680516 30295 catalog_manager.cc:4841] TS f996423f570146bdb87f10a69d734d2c (127.15.134.195:44605): tablet 2129daeb3c394d96a0a94e36bfa2d21d (table test-table [id=ca962e0a28e447d2bd1cd2b9a79a89f6]) successfully deleted
I20260928 08:33:48.680936 30295 catalog_manager.cc:4841] TS 84a23a8f68e047458b7932ff1803136b (127.15.134.194:46523): tablet 2129daeb3c394d96a0a94e36bfa2d21d (table test-table [id=ca962e0a28e447d2bd1cd2b9a79a89f6]) successfully deleted
I20260928 08:33:48.775213 30799 tablet_replica.cc:330] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac: stopping tablet replica
I20260928 08:33:48.775758 30799 raft_consensus.cc:2229] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:33:48.776496 30799 raft_consensus.cc:2258] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:33:48.777631 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 30348
W20260928 08:33:48.798033 30297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.134.193:33821 (error 108)
I20260928 08:33:48.798239 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 30500
I20260928 08:33:48.800168 30770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:48.799725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:33821 (local address 127.0.0.1:47120)
0928 08:33:48.799901 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:48.800013 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 08:33:48.800441 30297 catalog_manager.cc:4584] TS 28cdcfc4d9d14ccd91005a973261bfac (127.15.134.193:33821): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2129daeb3c394d96a0a94e36bfa2d21d: Network error: Client connection negotiation failed: client connection to 127.15.134.193:33821: connect: Connection refused (error 111)
I20260928 08:33:48.813021 30771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:48.799182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50450 (local address 127.15.134.194:46523)
0928 08:33:48.799379 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:48.799386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:48.799524 (+ 138us) server_negotiation.cc:407] Connection header received
0928 08:33:48.799739 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:48.799748 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:48.799821 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:48.799930 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:48.801467 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.802243 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.803627 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.804073 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.804220 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:48.807162 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:48.807178 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:48.807196 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:48.807233 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:48.810266 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:48.811036 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:48.811043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:48.811045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:48.811178 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:48.811536 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:48.811539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:48.811541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:48.811915 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:48.812142 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:48.812455 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 08:33:48.812689 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 08:33:48.813755 30623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:33:48.854359 30770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:48.853769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:33821 (local address 127.0.0.1:47124)
0928 08:33:48.854018 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:48.854132 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 08:33:48.855580 30770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:48.855148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:33821 (local address 127.0.0.1:47132)
0928 08:33:48.855331 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:48.855441 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 08:33:48.911011 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 30500
I20260928 08:33:48.916015 30770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:48.915546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:33821 (local address 127.0.0.1:47142)
0928 08:33:48.915756 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:48.915850 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 08:33:48.917389 30770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:48.916996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:33821 (local address 127.0.0.1:47156)
0928 08:33:48.917163 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:48.917269 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 08:33:48.933279 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 30631
I20260928 08:33:48.946961 30774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:48.934146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60986 (local address 127.15.134.195:44605)
0928 08:33:48.934337 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:48.934343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:48.934361 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:33:48.934463 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:48.934467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:48.934540 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:48.934629 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:48.936007 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.936860 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.938273 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:48.938651 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:48.938742 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:48.941616 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:48.941630 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:48.941648 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:48.941682 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:48.944287 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:48.944990 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:48.944997 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:48.945000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:48.945141 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:48.945616 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:48.945619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:48.945621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:48.945950 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:48.946181 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:48.946548 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 08:33:48.946711 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 08:33:48.947818 30754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:33:48.998201 30770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:48.997599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:33821 (local address 127.0.0.1:47166)
0928 08:33:48.997953 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:48.998041 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218}
I20260928 08:33:48.999233 30770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:48.998852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:33821 (local address 127.0.0.1:47168)
0928 08:33:48.999059 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:48.999127 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:33821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260928 08:33:49.038424 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 30631
I20260928 08:33:49.056671 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 30278
I20260928 08:33:49.057931 30341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:33:49.174790 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 30278
I20260928 08:33:49.197880 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:41481
--webserver_interface=127.15.134.254
--webserver_port=42229
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:44545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:41481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:33:49.539415 30807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:33:49.539724 30807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:33:49.539880 30807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:33:49.547349 30807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:33:49.547472 30807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:33:49.547562 30807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:33:49.547677 30807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:33:49.547740 30807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:33:49.558316 30807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:44545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:41481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:41481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=42229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:33:49.559904 30807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:33:49.561404 30807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:33:49.574115 30813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:49.574069 30815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:49.574167 30812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:49.578411 30807 server_base.cc:1034] running on GCE node
I20260928 08:33:49.579335 30807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:33:49.581032 30807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:33:49.582276 30807 hybrid_clock.cc:648] HybridClock initialized: now 1790584429582222 us; error 47 us; skew 500 ppm
I20260928 08:33:49.582651 30807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:33:49 dist-test-slave-qp2n krb5kdc[29726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584429, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:33:49.585950 30807 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:33:49.587637 30807 webserver.cc:460] Webserver started at http://127.15.134.254:42229/ using document root <none> and password file <none>
I20260928 08:33:49.588392 30807 fs_manager.cc:359] Metadata directory not provided
I20260928 08:33:49.588501 30807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:33:49.594496 30807 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 08:33:49.597726 30822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:49.599308 30807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:33:49.599499 30807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "16349924272d46a988a8e12fa7eb8360"
format_stamp: "Formatted at 2026-09-28 08:33:43 on dist-test-slave-qp2n"
server_key: "3fa15451eb890da685b9562e5ec9303c"
server_key_iv: "56738fcd9f8f0b7460f5fb5a07e5e2d1"
server_key_version: "encryptionkey@0"
I20260928 08:33:49.600579 30807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:33:49.635588 30807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:33:49.639904 30807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:33:49.640197 30807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:33:49.660215 30807 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:41481
I20260928 08:33:49.660249 30873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:41481 every 8 connection(s)
I20260928 08:33:49.661856 30807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:33:49.669473 30874 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:33:49.672226 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 30807
I20260928 08:33:49.673411 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:33821
--local_ip_for_outbound_sockets=127.15.134.193
--tserver_master_addrs=127.15.134.254:41481
--webserver_port=40457
--webserver_interface=127.15.134.193
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:44545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:33:49.673714 30874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360: Bootstrap starting.
I20260928 08:33:49.698618 30874 log.cc:824] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360: Log is configured to *not* fsync() on all Append() calls
I20260928 08:33:49.716795 30874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:33:49.717355 30874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360: Bootstrap complete.
I20260928 08:33:49.726761 30874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16349924272d46a988a8e12fa7eb8360" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41481 } }
I20260928 08:33:49.727653 30874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16349924272d46a988a8e12fa7eb8360, State: Initialized, Role: FOLLOWER
I20260928 08:33:49.728354 30874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16349924272d46a988a8e12fa7eb8360" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41481 } }
I20260928 08:33:49.728600 30874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:33:49.728737 30874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:33:49.728952 30874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [term 2 FOLLOWER]: Advancing to term 3
I20260928 08:33:49.733080 30874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16349924272d46a988a8e12fa7eb8360" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41481 } }
I20260928 08:33:49.733600 30874 leader_election.cc:302] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16349924272d46a988a8e12fa7eb8360; no voters:
I20260928 08:33:49.734380 30874 leader_election.cc:288] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 08:33:49.734511 30878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 08:33:49.735852 30878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [term 3 LEADER]: Becoming Leader. State: Replica: 16349924272d46a988a8e12fa7eb8360, State: Running, Role: LEADER
I20260928 08:33:49.736415 30878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16349924272d46a988a8e12fa7eb8360" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41481 } }
I20260928 08:33:49.736711 30874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:33:49.739006 30880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16349924272d46a988a8e12fa7eb8360. Latest consensus state: current_term: 3 leader_uuid: "16349924272d46a988a8e12fa7eb8360" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16349924272d46a988a8e12fa7eb8360" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41481 } } }
I20260928 08:33:49.739362 30880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [sys.catalog]: This master's current role is: LEADER
I20260928 08:33:49.748150 30879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "16349924272d46a988a8e12fa7eb8360" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16349924272d46a988a8e12fa7eb8360" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 41481 } } }
I20260928 08:33:49.748481 30879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16349924272d46a988a8e12fa7eb8360 [sys.catalog]: This master's current role is: LEADER
I20260928 08:33:49.749068 30883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:33:49.758291 30883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:33:49.759905 30883 catalog_manager.cc:1259] Loaded cluster ID: 0aaa7967c25b48dcbfdfa9ac3df6232e
I20260928 08:33:49.760069 30883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 08:33:49.762812 30883 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff2e7a259ad google::LogMessage::Fail() at ??:0
@ 0x7ff2e7a2888a google::LogMessage::SendToLog() at ??:0
@ 0x7ff2e7a254de google::LogMessage::Flush() at ??:0
@ 0x7ff2e7a26a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff2fd458512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ff2fd455976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff2fd4dd610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ff2fd4dd439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff2ea671345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff2ea67491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff2ea674799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff2ea64feaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff2ed9296db start_thread at ??:0
@ 0x7ff2e68bf61f clone at ??:0
W20260928 08:33:50.081406 30876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:33:50.081714 30876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:33:50.081885 30876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:33:50.090016 30876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:33:50.090157 30876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:33:50.090348 30876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:33:50.100347 30876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:44545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:33821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=40457
--tserver_master_addrs=127.15.134.254:41481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:33:50.101742 30876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:33:50.103103 30876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:33:50.116801 30918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:50.116909 30915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:50.116766 30916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:50.118067 30876 server_base.cc:1034] running on GCE node
I20260928 08:33:50.118671 30876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:33:50.119652 30876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:33:50.120893 30876 hybrid_clock.cc:648] HybridClock initialized: now 1790584430120835 us; error 49 us; skew 500 ppm
I20260928 08:33:50.121222 30876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:33:50 dist-test-slave-qp2n krb5kdc[29726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584430, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:33:50.128105 30876 init.cc:377] Logged in from keytab as kudu/127.15.134.193@KRBTEST.COM (short username kudu)
I20260928 08:33:50.129590 30876 webserver.cc:460] Webserver started at http://127.15.134.193:40457/ using document root <none> and password file <none>
I20260928 08:33:50.130313 30876 fs_manager.cc:359] Metadata directory not provided
I20260928 08:33:50.130461 30876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:33:50.136094 30876 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 08:33:50.139153 30925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:50.140594 30876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:33:50.140879 30876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "28cdcfc4d9d14ccd91005a973261bfac"
format_stamp: "Formatted at 2026-09-28 08:33:44 on dist-test-slave-qp2n"
server_key: "2ea04a7635602a12c19de0ca73dd8f59"
server_key_iv: "930d0735e018f10a56fb8bc660685917"
server_key_version: "encryptionkey@0"
I20260928 08:33:50.141911 30876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:33:50.157310 30876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:33:50.162564 30876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:33:50.162884 30876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:33:50.164036 30876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:33:50.166116 30932 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260928 08:33:50.172421 30876 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 08:33:50.172607 30876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260928 08:33:50.172834 30876 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 08:33:50.176441 30876 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 08:33:50.176566 30876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.000s sys 0.000s
I20260928 08:33:50.177204 30932 tablet_bootstrap.cc:492] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac: Bootstrap starting.
I20260928 08:33:50.187306 30932 log.cc:824] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac: Log is configured to *not* fsync() on all Append() calls
I20260928 08:33:50.204994 30932 tablet_bootstrap.cc:492] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:33:50.205691 30932 tablet_bootstrap.cc:492] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac: Bootstrap complete.
I20260928 08:33:50.206202 30932 ts_tablet_manager.cc:1397] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.005s
I20260928 08:33:50.215682 30932 raft_consensus.cc:348] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } } peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } }
I20260928 08:33:50.217329 30932 raft_consensus.cc:729] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28cdcfc4d9d14ccd91005a973261bfac, State: Initialized, Role: FOLLOWER
I20260928 08:33:50.218178 30932 consensus_queue.cc:260] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } } peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } }
I20260928 08:33:50.219931 30932 ts_tablet_manager.cc:1428] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260928 08:33:50.223376 30876 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:33821
I20260928 08:33:50.223450 31040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:33821 every 8 connection(s)
I20260928 08:33:50.224669 30876 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:33:50.229467 31043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:50.227489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.193:45947)
0928 08:33:50.228270 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:50.228514 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":321,"threads_started":1}
W20260928 08:33:50.230110 31041 heartbeater.cc:644] Failed to heartbeat to 127.15.134.254:41481 (0 consecutive failures): Network error: Failed to ping master at 127.15.134.254:41481: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
I20260928 08:33:50.230980 31043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:50.230636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.193:55657)
0928 08:33:50.230816 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:50.230882 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 08:33:50.231950 31043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:50.231493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.193:37859)
0928 08:33:50.231672 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:50.231783 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 08:33:50.232281 31041 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 08:33:50.232412 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 30876
I20260928 08:33:50.233069 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:46523
--local_ip_for_outbound_sockets=127.15.134.194
--tserver_master_addrs=127.15.134.254:41481
--webserver_port=36563
--webserver_interface=127.15.134.194
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:44545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:33:50.580061 31044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:33:50.580408 31044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:33:50.580595 31044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:33:50.588153 31044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:33:50.588308 31044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:33:50.588495 31044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:33:50.598829 31044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:44545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:46523
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=36563
--tserver_master_addrs=127.15.134.254:41481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:33:50.600232 31044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:33:50.601649 31044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:33:50.615173 31050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:50.615463 31052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:50.615217 31049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:50.615978 31044 server_base.cc:1034] running on GCE node
I20260928 08:33:50.616722 31044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:33:50.617684 31044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:33:50.618930 31044 hybrid_clock.cc:648] HybridClock initialized: now 1790584430618887 us; error 53 us; skew 500 ppm
I20260928 08:33:50.619269 31044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:33:50 dist-test-slave-qp2n krb5kdc[29726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584430, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:33:50.622514 31044 init.cc:377] Logged in from keytab as kudu/127.15.134.194@KRBTEST.COM (short username kudu)
I20260928 08:33:50.624037 31044 webserver.cc:460] Webserver started at http://127.15.134.194:36563/ using document root <none> and password file <none>
I20260928 08:33:50.624828 31044 fs_manager.cc:359] Metadata directory not provided
I20260928 08:33:50.624976 31044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:33:50.630992 31044 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 08:33:50.634087 31059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:50.635744 31044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:33:50.635965 31044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "84a23a8f68e047458b7932ff1803136b"
format_stamp: "Formatted at 2026-09-28 08:33:44 on dist-test-slave-qp2n"
server_key: "788e99bc59a6ee4d7bec42261b4a3fb2"
server_key_iv: "b3cc99c786929eccc3217f1290c8d219"
server_key_version: "encryptionkey@0"
I20260928 08:33:50.637249 31044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:33:50.659629 31044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:33:50.664130 31044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:33:50.664383 31044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:33:50.665400 31044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:33:50.666814 31044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:33:50.666903 31044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:50.666980 31044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:33:50.667035 31044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:50.702126 31044 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:46523
I20260928 08:33:50.702226 31171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:46523 every 8 connection(s)
I20260928 08:33:50.703691 31044 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:33:50.708755 31174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:50.706851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.194:60359)
0928 08:33:50.707588 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:50.707829 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":286,"threads_started":1}
I20260928 08:33:50.709383 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 31044
W20260928 08:33:50.709564 31172 heartbeater.cc:644] Failed to heartbeat to 127.15.134.254:41481 (0 consecutive failures): Network error: Failed to ping master at 127.15.134.254:41481: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
I20260928 08:33:50.710202 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:44605
--local_ip_for_outbound_sockets=127.15.134.195
--tserver_master_addrs=127.15.134.254:41481
--webserver_port=34823
--webserver_interface=127.15.134.195
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:44545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:33:50.710536 31174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:50.710136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.194:49359)
0928 08:33:50.710305 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:50.710393 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 08:33:50.711599 31174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:50.711185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.194:36097)
0928 08:33:50.711333 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:50.711439 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 08:33:50.711900 31172 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:33:51.046059 31175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:33:51.046350 31175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:33:51.046504 31175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:33:51.054121 31175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:33:51.054260 31175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:33:51.054456 31175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:33:51.064715 31175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:44545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:44605
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=34823
--tserver_master_addrs=127.15.134.254:41481
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:33:51.066080 31175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:33:51.067440 31175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:33:51.081933 31180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:51.081964 31181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:33:51.082204 31183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:33:51.082877 31175 server_base.cc:1034] running on GCE node
I20260928 08:33:51.083516 31175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:33:51.084451 31175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:33:51.085728 31175 hybrid_clock.cc:648] HybridClock initialized: now 1790584431085684 us; error 80 us; skew 500 ppm
I20260928 08:33:51.086086 31175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:33:51 dist-test-slave-qp2n krb5kdc[29726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584431, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:33:51.089478 31175 init.cc:377] Logged in from keytab as kudu/127.15.134.195@KRBTEST.COM (short username kudu)
I20260928 08:33:51.090979 31175 webserver.cc:460] Webserver started at http://127.15.134.195:34823/ using document root <none> and password file <none>
I20260928 08:33:51.091696 31175 fs_manager.cc:359] Metadata directory not provided
I20260928 08:33:51.091845 31175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:33:51.097362 31175 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:33:51.100154 31190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:51.101864 31175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:33:51.102079 31175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "f996423f570146bdb87f10a69d734d2c"
format_stamp: "Formatted at 2026-09-28 08:33:45 on dist-test-slave-qp2n"
server_key: "397a852c371f5c109bda49e7d3b9ef39"
server_key_iv: "74a5e53a3c35c6f34c1b47333d20504e"
server_key_version: "encryptionkey@0"
I20260928 08:33:51.103155 31175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:33:51.155798 31175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:33:51.160308 31175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:33:51.160600 31175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:33:51.161762 31175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:33:51.163190 31175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:33:51.163319 31175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:51.163450 31175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:33:51.163546 31175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:33:51.197846 31175 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:44605
I20260928 08:33:51.197956 31302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:44605 every 8 connection(s)
I20260928 08:33:51.199813 31175 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:33:51.205212 31305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:51.203226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.195:43309)
0928 08:33:51.203956 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:51.204220 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":278,"threads_started":1}
I20260928 08:33:51.205266 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 31175
W20260928 08:33:51.205955 31303 heartbeater.cc:644] Failed to heartbeat to 127.15.134.254:41481 (0 consecutive failures): Network error: Failed to ping master at 127.15.134.254:41481: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
I20260928 08:33:51.206844 31305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:51.206444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.195:51117)
0928 08:33:51.206600 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:51.206679 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 08:33:51.207870 31305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:51.207490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.195:51975)
0928 08:33:51.207674 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:51.207753 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 08:33:51.208225 31303 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 08:33:51.233563 31307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:51.232838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.193:55087)
0928 08:33:51.233258 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:51.233350 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":193,"threads_started":1}
I20260928 08:33:51.713370 31308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:51.712451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.194:35637)
0928 08:33:51.713004 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:51.713133 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":309,"threads_started":1}
I20260928 08:33:51.750284 31309 raft_consensus.cc:482] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:33:51.750629 31309 raft_consensus.cc:504] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } } peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } }
I20260928 08:33:51.752367 31309 leader_election.cc:288] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84a23a8f68e047458b7932ff1803136b (127.15.134.194:46523), f996423f570146bdb87f10a69d734d2c (127.15.134.195:44605)
Sep 28 08:33:51 dist-test-slave-qp2n krb5kdc[29726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584430, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for kudu/127.15.134.194@KRBTEST.COM
Sep 28 08:33:51 dist-test-slave-qp2n krb5kdc[29726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584430, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for kudu/127.15.134.195@KRBTEST.COM
I20260928 08:33:51.779363 31310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:51.752665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:46523 (local address 127.15.134.193:41741)
0928 08:33:51.753317 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:51.753350 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:51.754001 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:51.758134 (+ 4133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:51.758145 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:51.759124 (+ 979us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:33:51.759914 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:51.759946 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:51.762356 (+ 2410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:51.762360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:51.764573 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:51.764582 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:51.766246 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:51.766254 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:51.766695 (+ 441us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:51.767620 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:33:51.767674 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:33:51.771161 (+ 3487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:33:51.775398 (+ 4237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:51.775402 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:51.775404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:51.775865 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:51.776409 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:51.776412 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:51.776414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:51.776671 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:51.777620 (+ 949us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:33:51.777629 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:33:51.778000 (+ 371us) client_negotiation.cc:769] Sending connection context
0928 08:33:51.778263 (+ 263us) client_negotiation.cc:240] Negotiation successful
0928 08:33:51.778868 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":359,"spinlock_wait_cycles":100992,"thread_start_us":347,"threads_started":1}
I20260928 08:33:51.779350 31313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:51.754063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:46313 (local address 127.15.134.195:44605)
0928 08:33:51.754878 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:51.754898 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:51.755438 (+ 540us) server_negotiation.cc:407] Connection header received
0928 08:33:51.757158 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:51.757168 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:51.757604 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:51.757942 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:51.760240 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:51.762135 (+ 1895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:51.764790 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:51.765956 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:51.766181 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:51.771836 (+ 5655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:51.771843 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:51.771855 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:51.771892 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:51.775013 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:51.776057 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:51.776063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:51.776069 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:51.776246 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:51.776947 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:51.776951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:51.776953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:51.777465 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:51.777666 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:51.778538 (+ 872us) server_negotiation.cc:299] Negotiation successful
0928 08:33:51.778909 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":527,"threads_started":1}
I20260928 08:33:51.779552 31312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:51.753620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:44605 (local address 127.15.134.193:46313)
0928 08:33:51.755296 (+ 1676us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:51.755319 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:33:51.755474 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:33:51.758138 (+ 2664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:33:51.758151 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:33:51.759166 (+ 1015us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:33:51.760039 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:51.760057 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:51.762784 (+ 2727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:51.762789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:51.764561 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:33:51.764577 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:33:51.766247 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:33:51.766254 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:33:51.766693 (+ 439us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:51.767785 (+ 1092us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:33:51.767803 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:33:51.771585 (+ 3782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:33:51.775209 (+ 3624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:51.775219 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:51.775235 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:51.775887 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:51.776391 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:33:51.776397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:33:51.776408 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:33:51.776815 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:33:51.777627 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:33:51.777632 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:33:51.777998 (+ 366us) client_negotiation.cc:769] Sending connection context
0928 08:33:51.778276 (+ 278us) client_negotiation.cc:240] Negotiation successful
0928 08:33:51.779040 (+ 764us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1517,"mutex_wait_us":488,"spinlock_wait_cycles":32896,"thread_start_us":1297,"threads_started":1}
I20260928 08:33:51.779420 31311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:51.752886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:41741 (local address 127.15.134.194:46523)
0928 08:33:51.755463 (+ 2577us) server_negotiation.cc:206] Beginning negotiation
0928 08:33:51.755478 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 08:33:51.755527 (+ 49us) server_negotiation.cc:407] Connection header received
0928 08:33:51.757282 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:33:51.757291 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:33:51.757643 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:33:51.758004 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:33:51.760218 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:51.762135 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:51.764790 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:33:51.765962 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:33:51.766181 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:33:51.771575 (+ 5394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:33:51.771588 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:33:51.771608 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:33:51.771673 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:33:51.775236 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:51.776016 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:51.776026 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:51.776046 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:51.776245 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:33:51.776821 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:33:51.776827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:33:51.776830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:33:51.777456 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:33:51.777677 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 08:33:51.778546 (+ 869us) server_negotiation.cc:299] Negotiation successful
0928 08:33:51.778935 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2364,"thread_start_us":2174,"threads_started":1}
W20260928 08:33:51.783016 30926 leader_election.cc:341] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 84a23a8f68e047458b7932ff1803136b (127.15.134.194:46523): Not found: Tablet not found: 2129daeb3c394d96a0a94e36bfa2d21d
W20260928 08:33:51.783389 30926 leader_election.cc:341] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f996423f570146bdb87f10a69d734d2c (127.15.134.195:44605): Not found: Tablet not found: 2129daeb3c394d96a0a94e36bfa2d21d
I20260928 08:33:51.783502 30926 leader_election.cc:302] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28cdcfc4d9d14ccd91005a973261bfac; no voters: 84a23a8f68e047458b7932ff1803136b, f996423f570146bdb87f10a69d734d2c
I20260928 08:33:51.783915 31309 raft_consensus.cc:2726] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:33:52.209993 31314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:52.209048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.195:40941)
0928 08:33:52.209553 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:52.209743 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":242,"threads_started":1}
I20260928 08:33:52.235095 31312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:52.234630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.193:35399)
0928 08:33:52.234820 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:52.234932 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 08:33:52.715092 31315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:52.714379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.194:42411)
0928 08:33:52.714857 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:52.714937 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":283,"threads_started":1}
I20260928 08:33:53.211637 31316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:53.210904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.195:34403)
0928 08:33:53.211376 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:53.211460 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":261,"threads_started":1}
I20260928 08:33:53.236860 31317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:53.236022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.193:32927)
0928 08:33:53.236490 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:53.236580 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":230,"threads_started":1}
I20260928 08:33:53.716889 31318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:53.716113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.194:60607)
0928 08:33:53.716568 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:53.716682 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":235,"threads_started":1}
I20260928 08:33:53.888827 31319 raft_consensus.cc:482] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:33:53.889000 31319 raft_consensus.cc:504] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } } peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } }
I20260928 08:33:53.889581 31319 leader_election.cc:288] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84a23a8f68e047458b7932ff1803136b (127.15.134.194:46523), f996423f570146bdb87f10a69d734d2c (127.15.134.195:44605)
W20260928 08:33:53.890640 30926 leader_election.cc:341] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 84a23a8f68e047458b7932ff1803136b (127.15.134.194:46523): Not found: Tablet not found: 2129daeb3c394d96a0a94e36bfa2d21d
W20260928 08:33:53.890861 30926 leader_election.cc:341] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f996423f570146bdb87f10a69d734d2c (127.15.134.195:44605): Not found: Tablet not found: 2129daeb3c394d96a0a94e36bfa2d21d
I20260928 08:33:53.890933 30926 leader_election.cc:302] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28cdcfc4d9d14ccd91005a973261bfac; no voters: 84a23a8f68e047458b7932ff1803136b, f996423f570146bdb87f10a69d734d2c
I20260928 08:33:53.891171 31319 raft_consensus.cc:2726] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:33:54.213626 31320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:54.212772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.195:35349)
0928 08:33:54.213301 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:54.213385 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":280,"threads_started":1}
I20260928 08:33:54.239044 31321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:54.237876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.193:41551)
0928 08:33:54.238508 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:54.238811 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":289,"threads_started":1}
I20260928 08:33:54.718621 31322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:54.717820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.194:55479)
0928 08:33:54.718346 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:54.718430 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":266,"threads_started":1}
I20260928 08:33:55.215296 31323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:55.214572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.195:55501)
0928 08:33:55.215034 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:55.215127 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":228,"threads_started":1}
I20260928 08:33:55.241024 31324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:55.240043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.193:44193)
0928 08:33:55.240712 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:55.240820 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":333,"threads_started":1}
I20260928 08:33:55.720559 31325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:55.719695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.194:57513)
0928 08:33:55.720248 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:55.720331 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":300,"threads_started":1}
I20260928 08:33:55.725943 31326 raft_consensus.cc:482] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:33:55.726166 31326 raft_consensus.cc:504] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } } peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } }
I20260928 08:33:55.726857 31326 leader_election.cc:288] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84a23a8f68e047458b7932ff1803136b (127.15.134.194:46523), f996423f570146bdb87f10a69d734d2c (127.15.134.195:44605)
W20260928 08:33:55.727865 30926 leader_election.cc:341] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 84a23a8f68e047458b7932ff1803136b (127.15.134.194:46523): Not found: Tablet not found: 2129daeb3c394d96a0a94e36bfa2d21d
W20260928 08:33:55.728096 30926 leader_election.cc:341] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f996423f570146bdb87f10a69d734d2c (127.15.134.195:44605): Not found: Tablet not found: 2129daeb3c394d96a0a94e36bfa2d21d
I20260928 08:33:55.728230 30926 leader_election.cc:302] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28cdcfc4d9d14ccd91005a973261bfac; no voters: 84a23a8f68e047458b7932ff1803136b, f996423f570146bdb87f10a69d734d2c
I20260928 08:33:55.728523 31326 raft_consensus.cc:2726] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:33:56.217238 31327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:56.216319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.195:37741)
0928 08:33:56.216961 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:56.217064 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":399,"threads_started":1}
I20260928 08:33:56.242959 31328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:56.241992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.193:55427)
0928 08:33:56.242611 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:56.242747 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":315,"threads_started":1}
I20260928 08:33:56.722266 31329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:56.721494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.194:57169)
0928 08:33:56.721961 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:56.722077 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":239,"threads_started":1}
I20260928 08:33:57.218938 31330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:57.218217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.195:46967)
0928 08:33:57.218678 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:57.218764 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":238,"threads_started":1}
I20260928 08:33:57.245024 31331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:57.244038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.193:42737)
0928 08:33:57.244543 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:57.244811 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":288,"threads_started":1}
I20260928 08:33:57.724435 31332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:57.723447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.194:41871)
0928 08:33:57.724132 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:57.724225 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":314,"threads_started":1}
I20260928 08:33:58.180786 31333 raft_consensus.cc:482] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:33:58.181030 31333 raft_consensus.cc:504] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } } peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } }
I20260928 08:33:58.181753 31333 leader_election.cc:288] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84a23a8f68e047458b7932ff1803136b (127.15.134.194:46523), f996423f570146bdb87f10a69d734d2c (127.15.134.195:44605)
W20260928 08:33:58.182860 30926 leader_election.cc:341] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f996423f570146bdb87f10a69d734d2c (127.15.134.195:44605): Not found: Tablet not found: 2129daeb3c394d96a0a94e36bfa2d21d
W20260928 08:33:58.183104 30926 leader_election.cc:341] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 84a23a8f68e047458b7932ff1803136b (127.15.134.194:46523): Not found: Tablet not found: 2129daeb3c394d96a0a94e36bfa2d21d
I20260928 08:33:58.183295 30926 leader_election.cc:302] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28cdcfc4d9d14ccd91005a973261bfac; no voters: 84a23a8f68e047458b7932ff1803136b, f996423f570146bdb87f10a69d734d2c
I20260928 08:33:58.183660 31333 raft_consensus.cc:2726] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:33:58.221056 31334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:58.219997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.195:49401)
0928 08:33:58.220726 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:58.220828 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":256,"threads_started":1}
I20260928 08:33:58.246942 31335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:58.246044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.193:60291)
0928 08:33:58.246516 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:58.246704 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":250,"threads_started":1}
I20260928 08:33:58.726379 31336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:58.725583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.194:51199)
0928 08:33:58.726060 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:58.726190 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":237,"threads_started":1}
I20260928 08:33:59.222784 31337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:59.222082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.195:41793)
0928 08:33:59.222511 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:59.222595 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":221,"threads_started":1}
I20260928 08:33:59.248718 31338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:59.247911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.193:46757)
0928 08:33:59.248401 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:59.248481 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":247,"threads_started":1}
I20260928 08:33:59.728303 31339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:33:59.727424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.194:57087)
0928 08:33:59.728027 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0928 08:33:59.728129 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":280,"threads_started":1}
I20260928 08:34:00.224709 31340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:00.223798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.195:42079)
0928 08:34:00.224400 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:00.224487 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":384,"threads_started":1}
I20260928 08:34:00.250517 31341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:00.249750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.193:53827)
0928 08:34:00.250263 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:00.250355 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":253,"threads_started":1}
I20260928 08:34:00.730762 31342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:00.729705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.194:37069)
0928 08:34:00.730294 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:00.730404 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":313,"threads_started":1}
I20260928 08:34:01.226451 31343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:01.225650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.195:33441)
0928 08:34:01.226174 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:01.226270 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":295,"threads_started":1}
I20260928 08:34:01.252326 31344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:01.251532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.193:53325)
0928 08:34:01.252043 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:01.252138 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":270,"threads_started":1}
I20260928 08:34:01.732527 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:01.731759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.194:43543)
0928 08:34:01.732267 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:01.732363 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":242,"threads_started":1}
I20260928 08:34:02.228209 31346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:02.227478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.195:60723)
0928 08:34:02.227914 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:02.228014 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":227,"threads_started":1}
I20260928 08:34:02.254112 31347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:02.253360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.193:60303)
0928 08:34:02.253840 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:02.253930 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":249,"threads_started":1}
I20260928 08:34:02.721882 31348 raft_consensus.cc:482] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:34:02.722105 31348 raft_consensus.cc:504] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84a23a8f68e047458b7932ff1803136b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46523 } } peers { permanent_uuid: "28cdcfc4d9d14ccd91005a973261bfac" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33821 } } peers { permanent_uuid: "f996423f570146bdb87f10a69d734d2c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44605 } }
I20260928 08:34:02.722932 31348 leader_election.cc:288] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 84a23a8f68e047458b7932ff1803136b (127.15.134.194:46523), f996423f570146bdb87f10a69d734d2c (127.15.134.195:44605)
W20260928 08:34:02.724046 30926 leader_election.cc:341] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f996423f570146bdb87f10a69d734d2c (127.15.134.195:44605): Not found: Tablet not found: 2129daeb3c394d96a0a94e36bfa2d21d
W20260928 08:34:02.724289 30926 leader_election.cc:341] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 84a23a8f68e047458b7932ff1803136b (127.15.134.194:46523): Not found: Tablet not found: 2129daeb3c394d96a0a94e36bfa2d21d
I20260928 08:34:02.724426 30926 leader_election.cc:302] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 28cdcfc4d9d14ccd91005a973261bfac; no voters: 84a23a8f68e047458b7932ff1803136b, f996423f570146bdb87f10a69d734d2c
I20260928 08:34:02.724766 31348 raft_consensus.cc:2726] T 2129daeb3c394d96a0a94e36bfa2d21d P 28cdcfc4d9d14ccd91005a973261bfac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:34:02.734427 31349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:02.733553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.194:48445)
0928 08:34:02.734108 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:02.734228 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":271,"threads_started":1}
I20260928 08:34:03.229939 31350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:03.229156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.195:54269)
0928 08:34:03.229646 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:03.229738 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":259,"threads_started":1}
I20260928 08:34:03.255949 31351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:03.255212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.193:42537)
0928 08:34:03.255678 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:03.255773 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":246,"threads_started":1}
I20260928 08:34:03.736132 31352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:03.735350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.194:51761)
0928 08:34:03.735827 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:03.735928 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":254,"threads_started":1}
I20260928 08:34:04.231711 31353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:04.230887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.195:53119)
0928 08:34:04.231332 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:04.231423 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":233,"threads_started":1}
I20260928 08:34:04.258045 31354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:04.257171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.193:50489)
0928 08:34:04.257723 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:04.257830 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":244,"threads_started":1}
I20260928 08:34:04.737875 31355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:04.737161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.194:33205)
0928 08:34:04.737595 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:04.737680 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":220,"threads_started":1}
I20260928 08:34:05.233474 31356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:05.232741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.195:34607)
0928 08:34:05.233189 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:05.233272 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":244,"threads_started":1}
I20260928 08:34:05.260020 31357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:05.259253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.193:49569)
0928 08:34:05.259737 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:05.259838 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":218,"threads_started":1}
I20260928 08:34:05.739915 31358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:05.739122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.194:58739)
0928 08:34:05.739629 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:05.739739 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":248,"threads_started":1}
I20260928 08:34:06.213681 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 30876
I20260928 08:34:06.235616 31360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:06.234686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.195:37287)
0928 08:34:06.235223 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:06.235387 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":248,"threads_started":1}
I20260928 08:34:06.240417 31359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:06.214963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49280 (local address 127.15.134.193:33821)
0928 08:34:06.215955 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:06.215971 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:06.215999 (+ 28us) server_negotiation.cc:407] Connection header received
0928 08:34:06.216065 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:06.216072 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:06.216297 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:06.216517 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:34:06.219104 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:06.220524 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:06.224698 (+ 4174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:06.225628 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:06.225751 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:06.232132 (+ 6381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:34:06.232149 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:34:06.232170 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:34:06.232219 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:34:06.237164 (+ 4945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:06.238005 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:06.238015 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:06.238019 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:06.238188 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:06.238613 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:06.238626 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:06.238630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:06.239000 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:34:06.239318 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:06.239682 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 08:34:06.239980 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":825,"thread_start_us":706,"threads_started":1}
I20260928 08:34:06.242061 31036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:34:06.334674 31362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:06.333552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:41481 (local address 127.15.134.193:44603)
0928 08:34:06.334091 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:06.334335 (+ 244us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:41481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":269,"threads_started":1}
I20260928 08:34:06.334764 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 30876
I20260928 08:34:06.354736 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 31044
I20260928 08:34:06.368911 31363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:06.355545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48986 (local address 127.15.134.194:46523)
0928 08:34:06.356154 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:06.356158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:06.356174 (+ 16us) server_negotiation.cc:407] Connection header received
0928 08:34:06.356238 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:06.356242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:06.356300 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:06.356393 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:34:06.357793 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:06.358618 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:06.359869 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:06.360328 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:06.360430 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:06.363316 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:34:06.363321 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:34:06.363324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:34:06.363361 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:34:06.366329 (+ 2968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:06.367053 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:06.367058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:06.367060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:06.367197 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:06.367549 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:06.367553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:06.367555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:06.367830 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:34:06.368057 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:06.368294 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 08:34:06.368544 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":375,"threads_started":1}
I20260928 08:34:06.369668 31167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:34:06.458964 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 31044
I20260928 08:34:06.478662 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 31175
I20260928 08:34:06.493574 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:06.479557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42578 (local address 127.15.134.195:44605)
0928 08:34:06.480280 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:06.480284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:06.480302 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:34:06.480359 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:06.480363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:06.480425 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:06.480522 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:34:06.482061 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:06.482844 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:06.484107 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:06.484592 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:06.484835 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:06.487579 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:34:06.487584 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:34:06.487588 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:34:06.487624 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:34:06.490863 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:06.491604 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:06.491608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:06.491611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:06.491787 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:06.492173 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:06.492176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:06.492178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:06.492462 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:34:06.492815 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:06.492992 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 08:34:06.493174 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":454,"threads_started":1}
I20260928 08:34:06.494444 31298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:34:06.580318 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 31175
2026-09-28T08:34:06Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24856 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:34:06 dist-test-slave-qp2n krb5kdc[31369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:34:06 dist-test-slave-qp2n krb5kdc[31369](info): set up 2 sockets
Sep 28 08:34:06 dist-test-slave-qp2n krb5kdc[31369](info): commencing operation
krb5kdc: starting...
W20260928 08:34:07.746788 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:34:07 dist-test-slave-qp2n krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:34:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:34:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:34:08.001068 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:46653
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:34745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:46653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:34:08.355360 31382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:34:08.355664 31382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:34:08.355823 31382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:34:08.363315 31382 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 08:34:08.363480 31382 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 08:34:08.363577 31382 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 08:34:08.363669 31382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:34:08.363775 31382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:34:08.363897 31382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:34:08.363991 31382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:34:08.364069 31382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:34:08.374558 31382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:34745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:46653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:46653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:34:08.375969 31382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:34:08.377545 31382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:34:08.389765 31387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:34:08.389902 31390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:34:08.390048 31388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:34:08.390930 31382 server_base.cc:1034] running on GCE node
I20260928 08:34:08.391880 31382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:34:08.393494 31382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:34:08.394764 31382 hybrid_clock.cc:648] HybridClock initialized: now 1790584448394703 us; error 36 us; skew 500 ppm
I20260928 08:34:08.395120 31382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:34:08 dist-test-slave-qp2n krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584448, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:34:08.398362 31382 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:34:08.399943 31382 webserver.cc:460] Webserver started at http://127.15.134.254:35445/ using document root <none> and password file <none>
I20260928 08:34:08.400767 31382 fs_manager.cc:359] Metadata directory not provided
I20260928 08:34:08.400902 31382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:34:08.401244 31382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:34:08.404338 31382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "200c54a5f2444d2eb26bd5b75625071b"
format_stamp: "Formatted at 2026-09-28 08:34:08 on dist-test-slave-qp2n"
server_key: "4552dde4d3458b7cfbdb36fb58f2350a"
server_key_iv: "28e6e492903605f7a9dd1a8225745b45"
server_key_version: "encryptionkey@0"
I20260928 08:34:08.405181 31382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "200c54a5f2444d2eb26bd5b75625071b"
format_stamp: "Formatted at 2026-09-28 08:34:08 on dist-test-slave-qp2n"
server_key: "4552dde4d3458b7cfbdb36fb58f2350a"
server_key_iv: "28e6e492903605f7a9dd1a8225745b45"
server_key_version: "encryptionkey@0"
I20260928 08:34:08.410617 31382 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:34:08.414413 31398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:08.415840 31382 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:34:08.416039 31382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "200c54a5f2444d2eb26bd5b75625071b"
format_stamp: "Formatted at 2026-09-28 08:34:08 on dist-test-slave-qp2n"
server_key: "4552dde4d3458b7cfbdb36fb58f2350a"
server_key_iv: "28e6e492903605f7a9dd1a8225745b45"
server_key_version: "encryptionkey@0"
I20260928 08:34:08.416270 31382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:34:08.442523 31382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:34:08.447278 31382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:34:08.447603 31382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:34:08.467487 31382 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:46653
I20260928 08:34:08.467509 31449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:46653 every 8 connection(s)
I20260928 08:34:08.469061 31382 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:34:08.473273 31450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:34:08.475860 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 31382
I20260928 08:34:08.476090 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:34:08.476956 15899 external_mini_cluster.cc:1428] Setting key 6f78f7cef96fa156d1f11cd172d81f20
I20260928 08:34:08.486928 31450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b: Bootstrap starting.
I20260928 08:34:08.491217 31450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b: Neither blocks nor log segments found. Creating new log.
I20260928 08:34:08.492769 31450 log.cc:824] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b: Log is configured to *not* fsync() on all Append() calls
Sep 28 08:34:08 dist-test-slave-qp2n krb5kdc[31369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:34:08.495954 31450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b: No bootstrap required, opened a new log
I20260928 08:34:08.503268 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:08.479389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40972 (local address 127.15.134.254:46653)
0928 08:34:08.480368 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:08.480382 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:08.480436 (+ 54us) server_negotiation.cc:407] Connection header received
0928 08:34:08.482438 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:08.482465 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:08.482935 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:08.483337 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:34:08.485343 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:08.487012 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:08.489113 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:08.489891 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:08.490050 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:08.495327 (+ 5277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:34:08.495335 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:34:08.495352 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:34:08.495390 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:34:08.498742 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:08.499560 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:08.499567 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:08.499574 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:08.499750 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:08.500203 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:08.500206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:08.500208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:08.500905 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:34:08.501261 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:08.501701 (+ 440us) server_negotiation.cc:299] Negotiation successful
0928 08:34:08.502173 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":463,"threads_started":1}
I20260928 08:34:08.504423 31450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200c54a5f2444d2eb26bd5b75625071b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46653 } }
I20260928 08:34:08.504944 31450 raft_consensus.cc:374] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:34:08.505076 31450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 200c54a5f2444d2eb26bd5b75625071b, State: Initialized, Role: FOLLOWER
I20260928 08:34:08.505729 31450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200c54a5f2444d2eb26bd5b75625071b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46653 } }
I20260928 08:34:08.506006 31450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:34:08.506173 31450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:34:08.506361 31450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:34:08.508656 31450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200c54a5f2444d2eb26bd5b75625071b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46653 } }
I20260928 08:34:08.509176 31450 leader_election.cc:302] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 200c54a5f2444d2eb26bd5b75625071b; no voters:
I20260928 08:34:08.509783 31450 leader_election.cc:288] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:34:08.509971 31455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:34:08.511224 31455 raft_consensus.cc:686] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [term 1 LEADER]: Becoming Leader. State: Replica: 200c54a5f2444d2eb26bd5b75625071b, State: Running, Role: LEADER
I20260928 08:34:08.511763 31455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200c54a5f2444d2eb26bd5b75625071b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46653 } }
I20260928 08:34:08.512075 31450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:34:08.514783 31457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 200c54a5f2444d2eb26bd5b75625071b. Latest consensus state: current_term: 1 leader_uuid: "200c54a5f2444d2eb26bd5b75625071b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200c54a5f2444d2eb26bd5b75625071b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46653 } } }
I20260928 08:34:08.515202 31457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [sys.catalog]: This master's current role is: LEADER
I20260928 08:34:08.515923 31456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "200c54a5f2444d2eb26bd5b75625071b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200c54a5f2444d2eb26bd5b75625071b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46653 } } }
I20260928 08:34:08.516474 31456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [sys.catalog]: This master's current role is: LEADER
I20260928 08:34:08.517942 31464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:34:08.523675 31464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:34:08.532648 31464 catalog_manager.cc:1347] Generated new cluster ID: 9c6dad54183047e6bc3504556dc7e5fb
I20260928 08:34:08.532770 31464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:34:08.553211 31464 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:34:08.555454 31464 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:34:08.565899 31464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b: Generated new TSK 0
I20260928 08:34:08.567158 31464 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.134.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
I20260928 08:34:08.632718 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:0
--local_ip_for_outbound_sockets=127.15.134.193
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:46653
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:34745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:34:08.974018 31476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:34:08.974328 31476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:34:08.974427 31476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:34:08.981983 31476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:34:08.982107 31476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:34:08.982226 31476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:34:08.992466 31476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:34745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:46653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:34:08.994004 31476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:34:08.995507 31476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:34:09.009694 31481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:34:09.009756 31482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:34:09.009756 31484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:34:09.010829 31476 server_base.cc:1034] running on GCE node
I20260928 08:34:09.011453 31476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:34:09.012539 31476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:34:09.013736 31476 hybrid_clock.cc:648] HybridClock initialized: now 1790584449013695 us; error 32 us; skew 500 ppm
I20260928 08:34:09.014065 31476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:34:09 dist-test-slave-qp2n krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584449, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:34:09.017603 31476 init.cc:377] Logged in from keytab as kudu/127.15.134.193@KRBTEST.COM (short username kudu)
I20260928 08:34:09.019450 31476 webserver.cc:460] Webserver started at http://127.15.134.193:35381/ using document root <none> and password file <none>
I20260928 08:34:09.020462 31476 fs_manager.cc:359] Metadata directory not provided
I20260928 08:34:09.020674 31476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:34:09.021003 31476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:34:09.024065 31476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/data/instance:
uuid: "ee2bdf94afca4a05bc4a120205a1bb6e"
format_stamp: "Formatted at 2026-09-28 08:34:09 on dist-test-slave-qp2n"
server_key: "72c8ee8b8a5128348b43fdb6d020dba8"
server_key_iv: "1a954c4ac128359b52ba7158345e4ec0"
server_key_version: "encryptionkey@0"
I20260928 08:34:09.024868 31476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance:
uuid: "ee2bdf94afca4a05bc4a120205a1bb6e"
format_stamp: "Formatted at 2026-09-28 08:34:09 on dist-test-slave-qp2n"
server_key: "72c8ee8b8a5128348b43fdb6d020dba8"
server_key_iv: "1a954c4ac128359b52ba7158345e4ec0"
server_key_version: "encryptionkey@0"
I20260928 08:34:09.030246 31476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:34:09.033926 31491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:09.035573 31476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:34:09.035811 31476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "ee2bdf94afca4a05bc4a120205a1bb6e"
format_stamp: "Formatted at 2026-09-28 08:34:09 on dist-test-slave-qp2n"
server_key: "72c8ee8b8a5128348b43fdb6d020dba8"
server_key_iv: "1a954c4ac128359b52ba7158345e4ec0"
server_key_version: "encryptionkey@0"
I20260928 08:34:09.036010 31476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:34:09.077404 31476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:34:09.082052 31476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:34:09.082327 31476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:34:09.083284 31476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:34:09.084873 31476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:34:09.084949 31476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:09.085038 31476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:34:09.085083 31476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:09.120095 31476 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:43493
I20260928 08:34:09.120195 31603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:43493 every 8 connection(s)
I20260928 08:34:09.121493 31476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:34:09.127748 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 31476
I20260928 08:34:09.128027 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance
I20260928 08:34:09.128945 15899 external_mini_cluster.cc:1428] Setting key 58e2c4a1a07b021ea169d79cfa0af182
Sep 28 08:34:09 dist-test-slave-qp2n krb5kdc[31369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584449, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:34:09.145910 31606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:09.124215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:40967 (local address 127.15.134.254:46653)
0928 08:34:09.124720 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:09.124725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:09.125619 (+ 894us) server_negotiation.cc:407] Connection header received
0928 08:34:09.127462 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:09.127468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:09.127564 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:09.127698 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:34:09.129490 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:09.130552 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:09.133243 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:09.133695 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:09.133812 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:09.138555 (+ 4743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:34:09.138565 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:34:09.138573 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:34:09.138625 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:34:09.142213 (+ 3588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:09.143095 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:09.143103 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:09.143110 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:09.143270 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:09.143729 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:09.143736 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:09.143742 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:09.144167 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:34:09.144334 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:09.145155 (+ 821us) server_negotiation.cc:299] Negotiation successful
0928 08:34:09.145420 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":262,"threads_started":1}
I20260928 08:34:09.146613 31607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:09.124700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.193:40967)
0928 08:34:09.125375 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:09.125416 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 08:34:09.127172 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:34:09.127974 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:34:09.127987 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:34:09.128587 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:34:09.129299 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:09.129321 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:09.130771 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:09.130780 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:09.133037 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:09.133062 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:09.133974 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:09.133983 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:09.134357 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:09.135228 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:34:09.135267 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:34:09.138225 (+ 2958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:34:09.142407 (+ 4182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:34:09.142416 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:34:09.142432 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:34:09.142914 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:34:09.143381 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:34:09.143384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:34:09.143387 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:34:09.143603 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:34:09.144296 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:34:09.144303 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:34:09.144714 (+ 411us) client_negotiation.cc:769] Sending connection context
0928 08:34:09.144970 (+ 256us) client_negotiation.cc:240] Negotiation successful
0928 08:34:09.145395 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":270,"threads_started":1}
I20260928 08:34:09.148319 31604 heartbeater.cc:347] Connected to a master server at 127.15.134.254:46653
I20260928 08:34:09.148695 31604 heartbeater.cc:464] Registering TS with master...
I20260928 08:34:09.149659 31604 heartbeater.cc:511] Master 127.15.134.254:46653 requested a full tablet report, sending...
I20260928 08:34:09.151973 31415 ts_manager.cc:194] Registered new tserver with Master: ee2bdf94afca4a05bc4a120205a1bb6e (127.15.134.193:43493)
I20260928 08:34:09.153597 31415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.193@KRBTEST.COM'} at 127.15.134.193:40967
WARNING: no policy specified for kudu/127.15.134.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
I20260928 08:34:09.188372 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:0
--local_ip_for_outbound_sockets=127.15.134.194
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:46653
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:34745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:34:09.528007 31610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:34:09.528288 31610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:34:09.528414 31610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:34:09.535477 31610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:34:09.535586 31610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:34:09.535696 31610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:34:09.545691 31610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:34745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:46653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:34:09.546972 31610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:34:09.548256 31610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:34:09.561348 31615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:34:09.561739 31616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:34:09.562253 31618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:34:09.563187 31610 server_base.cc:1034] running on GCE node
I20260928 08:34:09.563866 31610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:34:09.564913 31610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:34:09.566133 31610 hybrid_clock.cc:648] HybridClock initialized: now 1790584449566072 us; error 60 us; skew 500 ppm
I20260928 08:34:09.566449 31610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:34:09 dist-test-slave-qp2n krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584449, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:34:09.569630 31610 init.cc:377] Logged in from keytab as kudu/127.15.134.194@KRBTEST.COM (short username kudu)
I20260928 08:34:09.571151 31610 webserver.cc:460] Webserver started at http://127.15.134.194:43883/ using document root <none> and password file <none>
I20260928 08:34:09.571863 31610 fs_manager.cc:359] Metadata directory not provided
I20260928 08:34:09.572011 31610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:34:09.572321 31610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:34:09.575292 31610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/data/instance:
uuid: "418589322bf34fd9a0f83d7516d32051"
format_stamp: "Formatted at 2026-09-28 08:34:09 on dist-test-slave-qp2n"
server_key: "d6a8ffaf1c0d3f933b195b2a496bd25e"
server_key_iv: "cd0ad2cf6db0b35a77953ca261cef488"
server_key_version: "encryptionkey@0"
I20260928 08:34:09.576067 31610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance:
uuid: "418589322bf34fd9a0f83d7516d32051"
format_stamp: "Formatted at 2026-09-28 08:34:09 on dist-test-slave-qp2n"
server_key: "d6a8ffaf1c0d3f933b195b2a496bd25e"
server_key_iv: "cd0ad2cf6db0b35a77953ca261cef488"
server_key_version: "encryptionkey@0"
I20260928 08:34:09.581384 31610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 08:34:09.584877 31625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:09.586210 31610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:34:09.586372 31610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "418589322bf34fd9a0f83d7516d32051"
format_stamp: "Formatted at 2026-09-28 08:34:09 on dist-test-slave-qp2n"
server_key: "d6a8ffaf1c0d3f933b195b2a496bd25e"
server_key_iv: "cd0ad2cf6db0b35a77953ca261cef488"
server_key_version: "encryptionkey@0"
I20260928 08:34:09.586570 31610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:34:09.617971 31610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:34:09.622232 31610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:34:09.622531 31610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:34:09.623643 31610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:34:09.625051 31610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:34:09.625156 31610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:09.625303 31610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:34:09.625383 31610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:09.660473 31610 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:44873
I20260928 08:34:09.660676 31737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:44873 every 8 connection(s)
I20260928 08:34:09.661965 31610 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:34:09.672250 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 31610
I20260928 08:34:09.672461 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance
I20260928 08:34:09.673004 15899 external_mini_cluster.cc:1428] Setting key fc82d585362715b9113371006341f874
Sep 28 08:34:09 dist-test-slave-qp2n krb5kdc[31369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584449, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:34:09.685041 31740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:09.664926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:36565 (local address 127.15.134.254:46653)
0928 08:34:09.665370 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:09.665375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:09.666640 (+ 1265us) server_negotiation.cc:407] Connection header received
0928 08:34:09.668599 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:09.668606 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:09.668731 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:09.668859 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:34:09.670599 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:09.671316 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:09.673224 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:09.673833 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:09.673985 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:09.677808 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:34:09.677817 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:34:09.677825 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:34:09.677880 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:34:09.681523 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:09.682361 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:09.682369 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:09.682376 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:09.682541 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:09.683051 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:09.683058 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:09.683064 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:09.683455 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:34:09.683622 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:09.684495 (+ 873us) server_negotiation.cc:299] Negotiation successful
0928 08:34:09.684757 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":226,"threads_started":1}
I20260928 08:34:09.686673 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:09.665602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.194:36565)
0928 08:34:09.666365 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:09.666477 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0928 08:34:09.668283 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:34:09.669128 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:34:09.669137 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:34:09.669744 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:34:09.670390 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:09.670411 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:09.671534 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:09.671538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:09.673038 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:09.673051 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:09.674070 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:09.674076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:09.674361 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:09.675142 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:34:09.675178 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:34:09.677552 (+ 2374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:34:09.681682 (+ 4130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:34:09.681689 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:34:09.681703 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:34:09.682208 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:34:09.682678 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:34:09.682681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:34:09.682684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:34:09.682913 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:34:09.683621 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:34:09.683627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:34:09.684050 (+ 423us) client_negotiation.cc:769] Sending connection context
0928 08:34:09.684406 (+ 356us) client_negotiation.cc:240] Negotiation successful
0928 08:34:09.685064 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":312,"threads_started":1}
I20260928 08:34:09.688253 31738 heartbeater.cc:347] Connected to a master server at 127.15.134.254:46653
I20260928 08:34:09.688661 31738 heartbeater.cc:464] Registering TS with master...
I20260928 08:34:09.689507 31738 heartbeater.cc:511] Master 127.15.134.254:46653 requested a full tablet report, sending...
I20260928 08:34:09.691128 31415 ts_manager.cc:194] Registered new tserver with Master: 418589322bf34fd9a0f83d7516d32051 (127.15.134.194:44873)
I20260928 08:34:09.692044 31415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.194@KRBTEST.COM'} at 127.15.134.194:36565
WARNING: no policy specified for kudu/127.15.134.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
I20260928 08:34:09.732970 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:0
--local_ip_for_outbound_sockets=127.15.134.195
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:46653
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:34745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:34:10.070392 31744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:34:10.070677 31744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:34:10.070832 31744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:34:10.078397 31744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:34:10.078517 31744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:34:10.078724 31744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:34:10.089421 31744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:34745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:46653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:34:10.090931 31744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:34:10.092252 31744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:34:10.105798 31749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:34:10.105751 31750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:34:10.106237 31752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:34:10.106649 31744 server_base.cc:1034] running on GCE node
I20260928 08:34:10.107191 31744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:34:10.108080 31744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:34:10.109272 31744 hybrid_clock.cc:648] HybridClock initialized: now 1790584450109206 us; error 59 us; skew 500 ppm
I20260928 08:34:10.109638 31744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:34:10 dist-test-slave-qp2n krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584450, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:34:10.113055 31744 init.cc:377] Logged in from keytab as kudu/127.15.134.195@KRBTEST.COM (short username kudu)
I20260928 08:34:10.114698 31744 webserver.cc:460] Webserver started at http://127.15.134.195:45465/ using document root <none> and password file <none>
I20260928 08:34:10.115393 31744 fs_manager.cc:359] Metadata directory not provided
I20260928 08:34:10.115548 31744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:34:10.115808 31744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:34:10.118806 31744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/data/instance:
uuid: "a16c6b5ccc094350af081af313e102f3"
format_stamp: "Formatted at 2026-09-28 08:34:10 on dist-test-slave-qp2n"
server_key: "a2437f1368815a68053c4fb84734a1f3"
server_key_iv: "feb16b090e18bb2e3bb3e4a43ee0c910"
server_key_version: "encryptionkey@0"
I20260928 08:34:10.119517 31744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance:
uuid: "a16c6b5ccc094350af081af313e102f3"
format_stamp: "Formatted at 2026-09-28 08:34:10 on dist-test-slave-qp2n"
server_key: "a2437f1368815a68053c4fb84734a1f3"
server_key_iv: "feb16b090e18bb2e3bb3e4a43ee0c910"
server_key_version: "encryptionkey@0"
I20260928 08:34:10.125241 31744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 08:34:10.128773 31759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:10.130220 31744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 08:34:10.130416 31744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "a16c6b5ccc094350af081af313e102f3"
format_stamp: "Formatted at 2026-09-28 08:34:10 on dist-test-slave-qp2n"
server_key: "a2437f1368815a68053c4fb84734a1f3"
server_key_iv: "feb16b090e18bb2e3bb3e4a43ee0c910"
server_key_version: "encryptionkey@0"
I20260928 08:34:10.130638 31744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:34:10.157491 31604 heartbeater.cc:503] Master 127.15.134.254:46653 was elected leader, sending a full tablet report...
I20260928 08:34:10.172500 31744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:34:10.176992 31744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:34:10.177254 31744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:34:10.178201 31744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:34:10.179569 31744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:34:10.179642 31744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:10.179723 31744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:34:10.179774 31744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:10.214965 31744 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:35431
I20260928 08:34:10.215101 31871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:35431 every 8 connection(s)
I20260928 08:34:10.216347 31744 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:34:10.218613 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 31744
I20260928 08:34:10.218816 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance
I20260928 08:34:10.219503 15899 external_mini_cluster.cc:1428] Setting key 8869553942ab70422f1665926d1e8bd9
Sep 28 08:34:10 dist-test-slave-qp2n krb5kdc[31369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584450, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:34:10.238672 31874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:10.219304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:50901 (local address 127.15.134.254:46653)
0928 08:34:10.219959 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:10.219967 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:10.220941 (+ 974us) server_negotiation.cc:407] Connection header received
0928 08:34:10.223367 (+ 2426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:10.223373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:10.223563 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:10.223669 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:34:10.225292 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.226176 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.227664 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.228318 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.228490 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:10.231919 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:34:10.231924 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:34:10.231927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:34:10.231967 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:34:10.235595 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:10.236390 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:10.236394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:10.236396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:10.236569 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:10.237044 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:10.237048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:10.237050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:10.237330 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:34:10.237507 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:10.238161 (+ 654us) server_negotiation.cc:299] Negotiation successful
0928 08:34:10.238340 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":346,"threads_started":1}
I20260928 08:34:10.239624 31875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:10.219891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.195:50901)
0928 08:34:10.220750 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:10.220810 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 08:34:10.223128 (+ 2318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:34:10.223898 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:34:10.223906 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:34:10.224443 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:34:10.225123 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:10.225143 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.226319 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.226323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:10.227520 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:10.227532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.228521 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.228525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:10.228801 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:10.229444 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:34:10.229467 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:34:10.231709 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:34:10.235744 (+ 4035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:34:10.235756 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:34:10.235775 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:34:10.236259 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:34:10.236731 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:34:10.236734 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:34:10.236737 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:34:10.236930 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:34:10.237446 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:34:10.237452 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:34:10.237803 (+ 351us) client_negotiation.cc:769] Sending connection context
0928 08:34:10.238066 (+ 263us) client_negotiation.cc:240] Negotiation successful
0928 08:34:10.238473 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":323,"threads_started":1}
I20260928 08:34:10.241123 31872 heartbeater.cc:347] Connected to a master server at 127.15.134.254:46653
I20260928 08:34:10.241590 31872 heartbeater.cc:464] Registering TS with master...
I20260928 08:34:10.242406 31872 heartbeater.cc:511] Master 127.15.134.254:46653 requested a full tablet report, sending...
I20260928 08:34:10.243971 31415 ts_manager.cc:194] Registered new tserver with Master: a16c6b5ccc094350af081af313e102f3 (127.15.134.195:35431)
I20260928 08:34:10.244899 31415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.195@KRBTEST.COM'} at 127.15.134.195:50901
I20260928 08:34:10.248257 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:34:10.267575 31874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:10.253052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40976 (local address 127.15.134.254:46653)
0928 08:34:10.253223 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:10.253228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:10.253616 (+ 388us) server_negotiation.cc:407] Connection header received
0928 08:34:10.253753 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:10.253756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:10.253822 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:10.253937 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:34:10.255600 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.256393 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.257978 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.258437 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.258577 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:10.261358 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:34:10.261363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:34:10.261367 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:34:10.261415 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:34:10.264125 (+ 2710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:10.265048 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:10.265052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:10.265055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:10.265192 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:10.265677 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:10.265687 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:10.265689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:10.266038 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:34:10.266237 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:10.266725 (+ 488us) server_negotiation.cc:299] Negotiation successful
0928 08:34:10.266997 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 08:34:10.271811 31415 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:34:10.278759 31415 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:34:10.315488 31887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:10.291190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50206 (local address 127.15.134.193:43493)
0928 08:34:10.291865 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:10.291872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:10.303445 (+ 11573us) server_negotiation.cc:407] Connection header received
0928 08:34:10.305924 (+ 2479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:10.305938 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:10.306306 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:10.306497 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:34:10.307960 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.309627 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.312381 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.313994 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.314206 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:10.314374 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:10.314961 (+ 587us) server_negotiation.cc:299] Negotiation successful
0928 08:34:10.315142 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":369,"threads_started":1}
I20260928 08:34:10.315578 31886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:10.291144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:43493 (local address 127.0.0.1:50206)
0928 08:34:10.303237 (+ 12093us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:10.303290 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 08:34:10.305604 (+ 2314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:34:10.306858 (+ 1254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:34:10.306865 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:34:10.306901 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:34:10.307778 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:10.307792 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.309809 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.309814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:10.312188 (+ 2374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:10.312203 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.314494 (+ 2291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.314501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:10.314708 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:10.314760 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 08:34:10.314914 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 08:34:10.315122 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11736,"thread_start_us":16230,"threads_started":1}
I20260928 08:34:10.315951 31888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:10.291786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40786 (local address 127.15.134.194:44873)
0928 08:34:10.292301 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:10.292307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:10.307024 (+ 14717us) server_negotiation.cc:407] Connection header received
0928 08:34:10.307757 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:10.307765 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:10.308004 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:10.308181 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:34:10.309205 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.310717 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.313144 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.314590 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.314744 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:10.314905 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:10.315446 (+ 541us) server_negotiation.cc:299] Negotiation successful
0928 08:34:10.315625 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":301,"threads_started":1}
I20260928 08:34:10.316452 31889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:10.291180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:44873 (local address 127.0.0.1:40786)
0928 08:34:10.306896 (+ 15716us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:10.306913 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:34:10.307516 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:34:10.308459 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:34:10.308463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:34:10.308490 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:34:10.308989 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:10.309001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.310909 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.310920 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:10.312969 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:10.312981 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.315054 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.315061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:10.315228 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:10.315269 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 08:34:10.316043 (+ 774us) client_negotiation.cc:240] Negotiation successful
0928 08:34:10.316181 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15596,"mutex_wait_us":1,"thread_start_us":12316,"threads_started":1}
I20260928 08:34:10.317320 31885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:10.294169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:35431 (local address 127.0.0.1:48040)
0928 08:34:10.307644 (+ 13475us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:10.307663 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:34:10.307796 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:34:10.308712 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:34:10.308716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:34:10.308741 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:34:10.309302 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:10.309316 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.312319 (+ 3003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.312326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:10.314546 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:10.314558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.316647 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.316653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:10.316821 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:10.316851 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 08:34:10.316931 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 08:34:10.317048 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13334,"mutex_wait_us":37,"thread_start_us":12452,"threads_started":1}
I20260928 08:34:10.317994 31890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:10.294405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48040 (local address 127.15.134.195:35431)
0928 08:34:10.307354 (+ 12949us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:10.307366 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:10.307936 (+ 570us) server_negotiation.cc:407] Connection header received
0928 08:34:10.308037 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:10.308044 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:10.308256 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:10.308435 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:34:10.309632 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.312177 (+ 2545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.314687 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.316131 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.316317 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:10.316481 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:10.317625 (+ 1144us) server_negotiation.cc:299] Negotiation successful
0928 08:34:10.317743 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12763,"thread_start_us":12610,"threads_started":1}
I20260928 08:34:10.320127 31539 tablet_service.cc:1467] Processing CreateTablet for tablet 51749cf47b084ca9b37cd4b625e8f4c5 (DEFAULT_TABLE table=test-table [id=d8d20710f56449cc8a505320f012d21f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:34:10.320127 31673 tablet_service.cc:1467] Processing CreateTablet for tablet 51749cf47b084ca9b37cd4b625e8f4c5 (DEFAULT_TABLE table=test-table [id=d8d20710f56449cc8a505320f012d21f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:34:10.320809 31807 tablet_service.cc:1467] Processing CreateTablet for tablet 51749cf47b084ca9b37cd4b625e8f4c5 (DEFAULT_TABLE table=test-table [id=d8d20710f56449cc8a505320f012d21f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:34:10.321669 31539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51749cf47b084ca9b37cd4b625e8f4c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:34:10.321695 31673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51749cf47b084ca9b37cd4b625e8f4c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:34:10.322295 31807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51749cf47b084ca9b37cd4b625e8f4c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:34:10.333817 31891 tablet_bootstrap.cc:492] T 51749cf47b084ca9b37cd4b625e8f4c5 P a16c6b5ccc094350af081af313e102f3: Bootstrap starting.
I20260928 08:34:10.336535 31892 tablet_bootstrap.cc:492] T 51749cf47b084ca9b37cd4b625e8f4c5 P ee2bdf94afca4a05bc4a120205a1bb6e: Bootstrap starting.
I20260928 08:34:10.337731 31893 tablet_bootstrap.cc:492] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051: Bootstrap starting.
I20260928 08:34:10.337770 31891 tablet_bootstrap.cc:654] T 51749cf47b084ca9b37cd4b625e8f4c5 P a16c6b5ccc094350af081af313e102f3: Neither blocks nor log segments found. Creating new log.
I20260928 08:34:10.339340 31891 log.cc:824] T 51749cf47b084ca9b37cd4b625e8f4c5 P a16c6b5ccc094350af081af313e102f3: Log is configured to *not* fsync() on all Append() calls
I20260928 08:34:10.341109 31892 tablet_bootstrap.cc:654] T 51749cf47b084ca9b37cd4b625e8f4c5 P ee2bdf94afca4a05bc4a120205a1bb6e: Neither blocks nor log segments found. Creating new log.
I20260928 08:34:10.341234 31893 tablet_bootstrap.cc:654] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051: Neither blocks nor log segments found. Creating new log.
I20260928 08:34:10.342278 31893 log.cc:824] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051: Log is configured to *not* fsync() on all Append() calls
I20260928 08:34:10.342391 31892 log.cc:824] T 51749cf47b084ca9b37cd4b625e8f4c5 P ee2bdf94afca4a05bc4a120205a1bb6e: Log is configured to *not* fsync() on all Append() calls
I20260928 08:34:10.343364 31891 tablet_bootstrap.cc:492] T 51749cf47b084ca9b37cd4b625e8f4c5 P a16c6b5ccc094350af081af313e102f3: No bootstrap required, opened a new log
I20260928 08:34:10.343679 31891 ts_tablet_manager.cc:1397] T 51749cf47b084ca9b37cd4b625e8f4c5 P a16c6b5ccc094350af081af313e102f3: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 08:34:10.345309 31892 tablet_bootstrap.cc:492] T 51749cf47b084ca9b37cd4b625e8f4c5 P ee2bdf94afca4a05bc4a120205a1bb6e: No bootstrap required, opened a new log
I20260928 08:34:10.345592 31893 tablet_bootstrap.cc:492] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051: No bootstrap required, opened a new log
I20260928 08:34:10.345607 31892 ts_tablet_manager.cc:1397] T 51749cf47b084ca9b37cd4b625e8f4c5 P ee2bdf94afca4a05bc4a120205a1bb6e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 08:34:10.345845 31893 ts_tablet_manager.cc:1397] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 08:34:10.353850 31892 raft_consensus.cc:348] T 51749cf47b084ca9b37cd4b625e8f4c5 P ee2bdf94afca4a05bc4a120205a1bb6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2bdf94afca4a05bc4a120205a1bb6e" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 43493 } } peers { permanent_uuid: "418589322bf34fd9a0f83d7516d32051" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44873 } } peers { permanent_uuid: "a16c6b5ccc094350af081af313e102f3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 35431 } }
I20260928 08:34:10.354318 31892 raft_consensus.cc:374] T 51749cf47b084ca9b37cd4b625e8f4c5 P ee2bdf94afca4a05bc4a120205a1bb6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:34:10.354423 31892 raft_consensus.cc:729] T 51749cf47b084ca9b37cd4b625e8f4c5 P ee2bdf94afca4a05bc4a120205a1bb6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee2bdf94afca4a05bc4a120205a1bb6e, State: Initialized, Role: FOLLOWER
I20260928 08:34:10.355127 31892 consensus_queue.cc:260] T 51749cf47b084ca9b37cd4b625e8f4c5 P ee2bdf94afca4a05bc4a120205a1bb6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2bdf94afca4a05bc4a120205a1bb6e" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 43493 } } peers { permanent_uuid: "418589322bf34fd9a0f83d7516d32051" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44873 } } peers { permanent_uuid: "a16c6b5ccc094350af081af313e102f3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 35431 } }
I20260928 08:34:10.354971 31891 raft_consensus.cc:348] T 51749cf47b084ca9b37cd4b625e8f4c5 P a16c6b5ccc094350af081af313e102f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2bdf94afca4a05bc4a120205a1bb6e" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 43493 } } peers { permanent_uuid: "418589322bf34fd9a0f83d7516d32051" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44873 } } peers { permanent_uuid: "a16c6b5ccc094350af081af313e102f3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 35431 } }
I20260928 08:34:10.355428 31891 raft_consensus.cc:374] T 51749cf47b084ca9b37cd4b625e8f4c5 P a16c6b5ccc094350af081af313e102f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:34:10.355532 31891 raft_consensus.cc:729] T 51749cf47b084ca9b37cd4b625e8f4c5 P a16c6b5ccc094350af081af313e102f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a16c6b5ccc094350af081af313e102f3, State: Initialized, Role: FOLLOWER
I20260928 08:34:10.356169 31891 consensus_queue.cc:260] T 51749cf47b084ca9b37cd4b625e8f4c5 P a16c6b5ccc094350af081af313e102f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2bdf94afca4a05bc4a120205a1bb6e" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 43493 } } peers { permanent_uuid: "418589322bf34fd9a0f83d7516d32051" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44873 } } peers { permanent_uuid: "a16c6b5ccc094350af081af313e102f3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 35431 } }
I20260928 08:34:10.357414 31892 ts_tablet_manager.cc:1428] T 51749cf47b084ca9b37cd4b625e8f4c5 P ee2bdf94afca4a05bc4a120205a1bb6e: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260928 08:34:10.356552 31893 raft_consensus.cc:348] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2bdf94afca4a05bc4a120205a1bb6e" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 43493 } } peers { permanent_uuid: "418589322bf34fd9a0f83d7516d32051" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44873 } } peers { permanent_uuid: "a16c6b5ccc094350af081af313e102f3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 35431 } }
I20260928 08:34:10.357616 31893 raft_consensus.cc:374] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:34:10.357738 31893 raft_consensus.cc:729] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418589322bf34fd9a0f83d7516d32051, State: Initialized, Role: FOLLOWER
I20260928 08:34:10.358556 31872 heartbeater.cc:503] Master 127.15.134.254:46653 was elected leader, sending a full tablet report...
I20260928 08:34:10.359202 31891 ts_tablet_manager.cc:1428] T 51749cf47b084ca9b37cd4b625e8f4c5 P a16c6b5ccc094350af081af313e102f3: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260928 08:34:10.359656 31893 consensus_queue.cc:260] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2bdf94afca4a05bc4a120205a1bb6e" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 43493 } } peers { permanent_uuid: "418589322bf34fd9a0f83d7516d32051" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44873 } } peers { permanent_uuid: "a16c6b5ccc094350af081af313e102f3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 35431 } }
I20260928 08:34:10.361012 31738 heartbeater.cc:503] Master 127.15.134.254:46653 was elected leader, sending a full tablet report...
I20260928 08:34:10.361722 31893 ts_tablet_manager.cc:1428] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
W20260928 08:34:10.374228 31605 tablet.cc:2369] T 51749cf47b084ca9b37cd4b625e8f4c5 P ee2bdf94afca4a05bc4a120205a1bb6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:34:10.392838 31899 raft_consensus.cc:482] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:34:10.393162 31899 raft_consensus.cc:504] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2bdf94afca4a05bc4a120205a1bb6e" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 43493 } } peers { permanent_uuid: "418589322bf34fd9a0f83d7516d32051" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44873 } } peers { permanent_uuid: "a16c6b5ccc094350af081af313e102f3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 35431 } }
I20260928 08:34:10.395017 31899 leader_election.cc:288] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee2bdf94afca4a05bc4a120205a1bb6e (127.15.134.193:43493), a16c6b5ccc094350af081af313e102f3 (127.15.134.195:35431)
I20260928 08:34:10.402678 31887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:10.395514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:46067 (local address 127.15.134.193:43493)
0928 08:34:10.395785 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:10.395793 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:10.396564 (+ 771us) server_negotiation.cc:407] Connection header received
0928 08:34:10.396892 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:10.396898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:10.396997 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:10.397085 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:34:10.397798 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.398624 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.400766 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.401605 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.401763 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:10.401854 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:10.402289 (+ 435us) server_negotiation.cc:299] Negotiation successful
0928 08:34:10.402414 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 08:34:10.402689 31890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:10.395569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:42937 (local address 127.15.134.195:35431)
0928 08:34:10.395797 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:10.395805 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:10.396795 (+ 990us) server_negotiation.cc:407] Connection header received
0928 08:34:10.397036 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:10.397040 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:10.397139 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:10.397234 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:34:10.397876 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.398577 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.400884 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.401629 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.401817 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:10.401888 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:10.402197 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 08:34:10.402349 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 08:34:10.402848 31900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:10.395204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:43493 (local address 127.15.134.194:46067)
0928 08:34:10.396447 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:10.396464 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:34:10.396700 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:34:10.397130 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:34:10.397134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:34:10.397158 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:34:10.397632 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:10.397646 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.398783 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.398787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:10.400505 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:10.400517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.401839 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.401846 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:10.402083 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:10.402107 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 08:34:10.402222 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 08:34:10.402400 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1102,"mutex_wait_us":115,"spinlock_wait_cycles":28544,"thread_start_us":870,"threads_started":1}
I20260928 08:34:10.403220 31901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:10.395700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:35431 (local address 127.15.134.194:42937)
0928 08:34:10.396602 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:10.396701 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0928 08:34:10.396855 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:34:10.397284 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:34:10.397288 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:34:10.397323 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:34:10.397694 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:10.397727 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.398732 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.398741 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:10.400724 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:10.400747 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.401885 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.401893 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:10.402042 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:10.402066 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 08:34:10.402140 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 08:34:10.402340 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":46,"thread_start_us":519,"threads_started":1}
I20260928 08:34:10.403980 31559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51749cf47b084ca9b37cd4b625e8f4c5" candidate_uuid: "418589322bf34fd9a0f83d7516d32051" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee2bdf94afca4a05bc4a120205a1bb6e" is_pre_election: true
I20260928 08:34:10.404160 31827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51749cf47b084ca9b37cd4b625e8f4c5" candidate_uuid: "418589322bf34fd9a0f83d7516d32051" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a16c6b5ccc094350af081af313e102f3" is_pre_election: true
I20260928 08:34:10.404644 31559 raft_consensus.cc:2454] T 51749cf47b084ca9b37cd4b625e8f4c5 P ee2bdf94afca4a05bc4a120205a1bb6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 418589322bf34fd9a0f83d7516d32051 in term 0.
I20260928 08:34:10.404644 31827 raft_consensus.cc:2454] T 51749cf47b084ca9b37cd4b625e8f4c5 P a16c6b5ccc094350af081af313e102f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 418589322bf34fd9a0f83d7516d32051 in term 0.
I20260928 08:34:10.405467 31627 leader_election.cc:302] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 418589322bf34fd9a0f83d7516d32051, a16c6b5ccc094350af081af313e102f3, ee2bdf94afca4a05bc4a120205a1bb6e; no voters:
I20260928 08:34:10.405917 31899 raft_consensus.cc:2781] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:34:10.406085 31899 raft_consensus.cc:482] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:34:10.406189 31899 raft_consensus.cc:3037] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:34:10.408455 31899 raft_consensus.cc:504] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2bdf94afca4a05bc4a120205a1bb6e" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 43493 } } peers { permanent_uuid: "418589322bf34fd9a0f83d7516d32051" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44873 } } peers { permanent_uuid: "a16c6b5ccc094350af081af313e102f3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 35431 } }
I20260928 08:34:10.409261 31899 leader_election.cc:288] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051 [CANDIDATE]: Term 1 election: Requested vote from peers ee2bdf94afca4a05bc4a120205a1bb6e (127.15.134.193:43493), a16c6b5ccc094350af081af313e102f3 (127.15.134.195:35431)
I20260928 08:34:10.409704 31559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51749cf47b084ca9b37cd4b625e8f4c5" candidate_uuid: "418589322bf34fd9a0f83d7516d32051" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee2bdf94afca4a05bc4a120205a1bb6e"
I20260928 08:34:10.409792 31827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51749cf47b084ca9b37cd4b625e8f4c5" candidate_uuid: "418589322bf34fd9a0f83d7516d32051" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a16c6b5ccc094350af081af313e102f3"
I20260928 08:34:10.409971 31559 raft_consensus.cc:3037] T 51749cf47b084ca9b37cd4b625e8f4c5 P ee2bdf94afca4a05bc4a120205a1bb6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:34:10.409988 31827 raft_consensus.cc:3037] T 51749cf47b084ca9b37cd4b625e8f4c5 P a16c6b5ccc094350af081af313e102f3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:34:10.413554 31827 raft_consensus.cc:2454] T 51749cf47b084ca9b37cd4b625e8f4c5 P a16c6b5ccc094350af081af313e102f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 418589322bf34fd9a0f83d7516d32051 in term 1.
I20260928 08:34:10.413558 31559 raft_consensus.cc:2454] T 51749cf47b084ca9b37cd4b625e8f4c5 P ee2bdf94afca4a05bc4a120205a1bb6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 418589322bf34fd9a0f83d7516d32051 in term 1.
I20260928 08:34:10.414022 31628 leader_election.cc:302] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 418589322bf34fd9a0f83d7516d32051, ee2bdf94afca4a05bc4a120205a1bb6e; no voters:
W20260928 08:34:10.414207 31739 tablet.cc:2369] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:34:10.414435 31899 raft_consensus.cc:2781] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:34:10.414947 31899 raft_consensus.cc:686] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051 [term 1 LEADER]: Becoming Leader. State: Replica: 418589322bf34fd9a0f83d7516d32051, State: Running, Role: LEADER
I20260928 08:34:10.415532 31899 consensus_queue.cc:237] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2bdf94afca4a05bc4a120205a1bb6e" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 43493 } } peers { permanent_uuid: "418589322bf34fd9a0f83d7516d32051" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44873 } } peers { permanent_uuid: "a16c6b5ccc094350af081af313e102f3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 35431 } }
I20260928 08:34:10.422955 31413 catalog_manager.cc:5495] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051 reported cstate change: term changed from 0 to 1, leader changed from <none> to 418589322bf34fd9a0f83d7516d32051 (127.15.134.194). New cstate: current_term: 1 leader_uuid: "418589322bf34fd9a0f83d7516d32051" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee2bdf94afca4a05bc4a120205a1bb6e" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 43493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "418589322bf34fd9a0f83d7516d32051" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a16c6b5ccc094350af081af313e102f3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 35431 } health_report { overall_health: UNKNOWN } } }
I20260928 08:34:10.447309 31888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:10.441455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40796 (local address 127.15.134.194:44873)
0928 08:34:10.441673 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:10.441679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:10.441782 (+ 103us) server_negotiation.cc:407] Connection header received
0928 08:34:10.442031 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:10.442044 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:10.442132 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:10.442289 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:34:10.442748 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.443490 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.445053 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.445489 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.445624 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:10.446159 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:34:10.446696 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:34:10.446862 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:10.446929 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 08:34:10.447016 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 08:34:10.456238 31827 raft_consensus.cc:1261] T 51749cf47b084ca9b37cd4b625e8f4c5 P a16c6b5ccc094350af081af313e102f3 [term 1 FOLLOWER]: Refusing update from remote peer 418589322bf34fd9a0f83d7516d32051: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:34:10.456238 31559 raft_consensus.cc:1261] T 51749cf47b084ca9b37cd4b625e8f4c5 P ee2bdf94afca4a05bc4a120205a1bb6e [term 1 FOLLOWER]: Refusing update from remote peer 418589322bf34fd9a0f83d7516d32051: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:34:10.457311 31902 consensus_queue.cc:1035] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a16c6b5ccc094350af081af313e102f3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:34:10.457659 31899 consensus_queue.cc:1035] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee2bdf94afca4a05bc4a120205a1bb6e" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 43493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 08:34:10.468044 31873 tablet.cc:2369] T 51749cf47b084ca9b37cd4b625e8f4c5 P a16c6b5ccc094350af081af313e102f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:34:10.474330 31907 mvcc.cc:205] Tried to move back new op lower bound from 7334233909058424832 to 7334233908906041344. Current Snapshot: MvccSnapshot[applied={T|T < 7334233909058424832}]
I20260928 08:34:10.487061 31415 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:34:10.487382 31415 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:34:10.493353 31415 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b: Sending DeleteTablet for 3 replicas of tablet 51749cf47b084ca9b37cd4b625e8f4c5
I20260928 08:34:10.495095 31539 tablet_service.cc:1514] Processing DeleteTablet for tablet 51749cf47b084ca9b37cd4b625e8f4c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:34:10 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:50206
I20260928 08:34:10.495926 31807 tablet_service.cc:1514] Processing DeleteTablet for tablet 51749cf47b084ca9b37cd4b625e8f4c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:34:10 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:48040
I20260928 08:34:10.496843 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 31476
I20260928 08:34:10.497746 31673 tablet_service.cc:1514] Processing DeleteTablet for tablet 51749cf47b084ca9b37cd4b625e8f4c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:34:10 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:40786
I20260928 08:34:10.498498 31806 tablet_service.cc:1514] Processing DeleteTablet for tablet 51749cf47b084ca9b37cd4b625e8f4c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:34:10 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:48040
I20260928 08:34:10.512586 31915 tablet_replica.cc:330] T 51749cf47b084ca9b37cd4b625e8f4c5 P a16c6b5ccc094350af081af313e102f3: stopping tablet replica
I20260928 08:34:10.513083 31915 raft_consensus.cc:2229] T 51749cf47b084ca9b37cd4b625e8f4c5 P a16c6b5ccc094350af081af313e102f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:34:10.513540 31915 raft_consensus.cc:2258] T 51749cf47b084ca9b37cd4b625e8f4c5 P a16c6b5ccc094350af081af313e102f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:34:10.517402 31916 tablet_replica.cc:330] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051: stopping tablet replica
I20260928 08:34:10.517827 31916 raft_consensus.cc:2229] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:34:10.518323 31914 tablet_replica.cc:330] T 51749cf47b084ca9b37cd4b625e8f4c5 P ee2bdf94afca4a05bc4a120205a1bb6e: stopping tablet replica
I20260928 08:34:10.518476 31916 raft_consensus.cc:2258] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:34:10.518747 31914 raft_consensus.cc:2229] T 51749cf47b084ca9b37cd4b625e8f4c5 P ee2bdf94afca4a05bc4a120205a1bb6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:34:10.519272 31914 raft_consensus.cc:2258] T 51749cf47b084ca9b37cd4b625e8f4c5 P ee2bdf94afca4a05bc4a120205a1bb6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:34:10.520860 31916 ts_tablet_manager.cc:1905] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:34:10.520943 31915 ts_tablet_manager.cc:1905] T 51749cf47b084ca9b37cd4b625e8f4c5 P a16c6b5ccc094350af081af313e102f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:34:10.526458 31914 ts_tablet_manager.cc:1905] T 51749cf47b084ca9b37cd4b625e8f4c5 P ee2bdf94afca4a05bc4a120205a1bb6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:34:10.527989 31915 ts_tablet_manager.cc:1918] T 51749cf47b084ca9b37cd4b625e8f4c5 P a16c6b5ccc094350af081af313e102f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:34:10.528163 31915 log.cc:1196] T 51749cf47b084ca9b37cd4b625e8f4c5 P a16c6b5ccc094350af081af313e102f3: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/51749cf47b084ca9b37cd4b625e8f4c5
I20260928 08:34:10.528658 31916 ts_tablet_manager.cc:1918] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:34:10.528780 31915 ts_tablet_manager.cc:1939] T 51749cf47b084ca9b37cd4b625e8f4c5 P a16c6b5ccc094350af081af313e102f3: Deleting consensus metadata
I20260928 08:34:10.528820 31916 log.cc:1196] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/51749cf47b084ca9b37cd4b625e8f4c5
I20260928 08:34:10.529335 31916 ts_tablet_manager.cc:1939] T 51749cf47b084ca9b37cd4b625e8f4c5 P 418589322bf34fd9a0f83d7516d32051: Deleting consensus metadata
I20260928 08:34:10.530755 31400 catalog_manager.cc:4841] TS a16c6b5ccc094350af081af313e102f3 (127.15.134.195:35431): tablet 51749cf47b084ca9b37cd4b625e8f4c5 (table test-table [id=d8d20710f56449cc8a505320f012d21f]) successfully deleted
W20260928 08:34:10.531407 31400 catalog_manager.cc:4808] TS a16c6b5ccc094350af081af313e102f3 (127.15.134.195:35431): delete failed for tablet 51749cf47b084ca9b37cd4b625e8f4c5 because the tablet was not found. No further retry: Not found: Tablet not found: 51749cf47b084ca9b37cd4b625e8f4c5
I20260928 08:34:10.532194 31399 catalog_manager.cc:4841] TS 418589322bf34fd9a0f83d7516d32051 (127.15.134.194:44873): tablet 51749cf47b084ca9b37cd4b625e8f4c5 (table test-table [id=d8d20710f56449cc8a505320f012d21f]) successfully deleted
Sep 28 08:34:10 dist-test-slave-qp2n krb5kdc[31369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.193@KRBTEST.COM
I20260928 08:34:10.536561 31914 ts_tablet_manager.cc:1918] T 51749cf47b084ca9b37cd4b625e8f4c5 P ee2bdf94afca4a05bc4a120205a1bb6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:34:10.536818 31914 log.cc:1196] T 51749cf47b084ca9b37cd4b625e8f4c5 P ee2bdf94afca4a05bc4a120205a1bb6e: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/wal/wals/51749cf47b084ca9b37cd4b625e8f4c5
I20260928 08:34:10.537385 31914 ts_tablet_manager.cc:1939] T 51749cf47b084ca9b37cd4b625e8f4c5 P ee2bdf94afca4a05bc4a120205a1bb6e: Deleting consensus metadata
I20260928 08:34:10.539285 31401 catalog_manager.cc:4841] TS ee2bdf94afca4a05bc4a120205a1bb6e (127.15.134.193:43493): tablet 51749cf47b084ca9b37cd4b625e8f4c5 (table test-table [id=d8d20710f56449cc8a505320f012d21f]) successfully deleted
I20260928 08:34:10.542901 31887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:10.499743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50210 (local address 127.15.134.193:43493)
0928 08:34:10.499928 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:10.499937 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:10.516859 (+ 16922us) server_negotiation.cc:407] Connection header received
0928 08:34:10.519127 (+ 2268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:10.519136 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:10.519224 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:10.519355 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:34:10.521859 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.522966 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.526697 (+ 3731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.527353 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.527450 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:10.535253 (+ 7803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:34:10.535270 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:34:10.535313 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:34:10.535363 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:34:10.539582 (+ 4219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:10.540446 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:10.540457 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:10.540460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:10.540659 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:10.541118 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:10.541123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:10.541127 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:10.541586 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:34:10.541913 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:10.542264 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 08:34:10.542511 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:34:10.543653 31599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:34:10.635658 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 31476
I20260928 08:34:10.654834 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 31610
Sep 28 08:34:10 dist-test-slave-qp2n krb5kdc[31369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.194@KRBTEST.COM
I20260928 08:34:10.674067 31888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:10.655652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40804 (local address 127.15.134.194:44873)
0928 08:34:10.655824 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:10.655831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:10.655904 (+ 73us) server_negotiation.cc:407] Connection header received
0928 08:34:10.656123 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:10.656128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:10.656194 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:10.656314 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:34:10.657725 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.658578 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.660133 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.661087 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.661221 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:10.666707 (+ 5486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:34:10.666720 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:34:10.666736 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:34:10.666786 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:34:10.670931 (+ 4145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:10.671780 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:10.671789 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:10.671793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:10.671950 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:10.672372 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:10.672377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:10.672381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:10.672819 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:34:10.673165 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:10.673458 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 08:34:10.673678 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 08:34:10.674808 31733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:34:10.792366 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 31610
I20260928 08:34:10.812500 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 31744
Sep 28 08:34:10 dist-test-slave-qp2n krb5kdc[31369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.195@KRBTEST.COM
I20260928 08:34:10.828189 31890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:10.813368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48050 (local address 127.15.134.195:35431)
0928 08:34:10.813658 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:10.813663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:10.813681 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:34:10.813739 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:10.813743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:10.813812 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:10.813901 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:34:10.815449 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.816387 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.817811 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:10.818204 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:10.818304 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:10.822906 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:34:10.822921 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:34:10.822940 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:34:10.822975 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:34:10.825849 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:10.826503 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:10.826509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:10.826511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:10.826639 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:10.826957 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:10.826960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:10.826962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:10.827311 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:34:10.827525 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:10.827761 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 08:34:10.827921 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260928 08:34:10.828892 31867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:34:10.921624 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 31744
I20260928 08:34:10.940418 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 31382
I20260928 08:34:10.941763 31445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:34:11.054132 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 31382
I20260928 08:34:31.077268 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:46653
--webserver_interface=127.15.134.254
--webserver_port=35445
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:34745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:46653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:34:31.424516 31922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:34:31.424973 31922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:34:31.425128 31922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:34:31.432729 31922 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 08:34:31.432866 31922 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 08:34:31.432972 31922 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 08:34:31.433096 31922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:34:31.433178 31922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:34:31.433286 31922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:34:31.433375 31922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:34:31.433455 31922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:34:31.444056 31922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:34745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:46653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:46653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=35445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:34:31.445544 31922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:34:31.447010 31922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:34:31.471101 31929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:34:31.471238 31931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:34:31.471675 31928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:34:31.471990 31922 server_base.cc:1034] running on GCE node
I20260928 08:34:31.472988 31922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:34:31.474604 31922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:34:31.475883 31922 hybrid_clock.cc:648] HybridClock initialized: now 1790584471475818 us; error 70 us; skew 500 ppm
I20260928 08:34:31.476300 31922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:34:31 dist-test-slave-qp2n krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584471, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:34:31.484202 31922 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:34:31.486461 31922 webserver.cc:460] Webserver started at http://127.15.134.254:35445/ using document root <none> and password file <none>
I20260928 08:34:31.487393 31922 fs_manager.cc:359] Metadata directory not provided
I20260928 08:34:31.487563 31922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:34:31.494508 31922 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 08:34:31.498385 31938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:31.500212 31922 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 08:34:31.500437 31922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "200c54a5f2444d2eb26bd5b75625071b"
format_stamp: "Formatted at 2026-09-28 08:34:08 on dist-test-slave-qp2n"
server_key: "4552dde4d3458b7cfbdb36fb58f2350a"
server_key_iv: "28e6e492903605f7a9dd1a8225745b45"
server_key_version: "encryptionkey@0"
I20260928 08:34:31.501772 31922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:34:31.521025 31922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:34:31.526129 31922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:34:31.526456 31922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:34:31.548336 31922 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:46653
I20260928 08:34:31.548347 31989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:46653 every 8 connection(s)
I20260928 08:34:31.550319 31922 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:34:31.553027 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 31922
I20260928 08:34:31.553952 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:43493
--local_ip_for_outbound_sockets=127.15.134.193
--tserver_master_addrs=127.15.134.254:46653
--webserver_port=35381
--webserver_interface=127.15.134.193
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:34745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:34:31.558250 31990 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:34:31.561800 31990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b: Bootstrap starting.
I20260928 08:34:31.592136 31990 log.cc:824] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:34:31.604758 31990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:34:31.605342 31990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b: Bootstrap complete.
I20260928 08:34:31.614964 31990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200c54a5f2444d2eb26bd5b75625071b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46653 } }
I20260928 08:34:31.615850 31990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 200c54a5f2444d2eb26bd5b75625071b, State: Initialized, Role: FOLLOWER
I20260928 08:34:31.616456 31990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200c54a5f2444d2eb26bd5b75625071b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46653 } }
I20260928 08:34:31.616731 31990 raft_consensus.cc:388] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:34:31.616878 31990 raft_consensus.cc:482] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:34:31.617058 31990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [term 1 FOLLOWER]: Advancing to term 2
I20260928 08:34:31.621065 31990 raft_consensus.cc:504] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200c54a5f2444d2eb26bd5b75625071b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46653 } }
I20260928 08:34:31.621762 31990 leader_election.cc:302] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 200c54a5f2444d2eb26bd5b75625071b; no voters:
I20260928 08:34:31.622427 31990 leader_election.cc:288] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 08:34:31.622565 31994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [term 2 FOLLOWER]: Leader election won for term 2
I20260928 08:34:31.623921 31994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [term 2 LEADER]: Becoming Leader. State: Replica: 200c54a5f2444d2eb26bd5b75625071b, State: Running, Role: LEADER
I20260928 08:34:31.624490 31994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200c54a5f2444d2eb26bd5b75625071b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46653 } }
I20260928 08:34:31.625001 31990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:34:31.627650 31995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "200c54a5f2444d2eb26bd5b75625071b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200c54a5f2444d2eb26bd5b75625071b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46653 } } }
I20260928 08:34:31.628022 31995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [sys.catalog]: This master's current role is: LEADER
I20260928 08:34:31.628417 31996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 200c54a5f2444d2eb26bd5b75625071b. Latest consensus state: current_term: 2 leader_uuid: "200c54a5f2444d2eb26bd5b75625071b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200c54a5f2444d2eb26bd5b75625071b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46653 } } }
I20260928 08:34:31.628767 31996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [sys.catalog]: This master's current role is: LEADER
I20260928 08:34:31.642005 32001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:34:31.649994 32001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:34:31.651525 32001 catalog_manager.cc:1259] Loaded cluster ID: 9c6dad54183047e6bc3504556dc7e5fb
I20260928 08:34:31.651629 32001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:34:31.655853 32001 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:34:31.657635 32001 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b: Loaded TSK: 0
I20260928 08:34:31.704219 32001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b: Generated new TSK 1
I20260928 08:34:31.704680 32001 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 08:34:31.958343 31992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:34:31.958648 31992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:34:31.958806 31992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:34:31.966290 31992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:34:31.966401 31992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:34:31.966572 31992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:34:31.976687 31992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:34745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:43493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=35381
--tserver_master_addrs=127.15.134.254:46653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:34:31.978117 31992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:34:31.979502 31992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:34:31.993064 32017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:34:31.993116 32018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:34:31.993953 31992 server_base.cc:1034] running on GCE node
W20260928 08:34:31.994091 32020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:34:31.994803 31992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:34:31.995849 31992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:34:31.997052 31992 hybrid_clock.cc:648] HybridClock initialized: now 1790584471996980 us; error 68 us; skew 500 ppm
I20260928 08:34:31.997465 31992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:34:31 dist-test-slave-qp2n krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584471, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:34:32.001029 31992 init.cc:377] Logged in from keytab as kudu/127.15.134.193@KRBTEST.COM (short username kudu)
I20260928 08:34:32.002921 31992 webserver.cc:460] Webserver started at http://127.15.134.193:35381/ using document root <none> and password file <none>
I20260928 08:34:32.003890 31992 fs_manager.cc:359] Metadata directory not provided
I20260928 08:34:32.004052 31992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:34:32.012044 31992 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:34:32.015655 32027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:32.017447 31992 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 08:34:32.017625 31992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "ee2bdf94afca4a05bc4a120205a1bb6e"
format_stamp: "Formatted at 2026-09-28 08:34:09 on dist-test-slave-qp2n"
server_key: "72c8ee8b8a5128348b43fdb6d020dba8"
server_key_iv: "1a954c4ac128359b52ba7158345e4ec0"
server_key_version: "encryptionkey@0"
I20260928 08:34:32.019001 31992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:34:32.035905 31992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:34:32.041429 31992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:34:32.041721 31992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:34:32.042918 31992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:34:32.044720 31992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:34:32.044840 31992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:32.044955 31992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:34:32.045027 31992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:32.086287 31992 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:43493
I20260928 08:34:32.086375 32139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:43493 every 8 connection(s)
I20260928 08:34:32.087761 31992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:34:32.095093 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 31992
I20260928 08:34:32.095888 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:44873
--local_ip_for_outbound_sockets=127.15.134.194
--tserver_master_addrs=127.15.134.254:46653
--webserver_port=43883
--webserver_interface=127.15.134.194
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:34745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:34:32 dist-test-slave-qp2n krb5kdc[31369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584471, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:34:32.115175 32143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:32.090962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.193:57629)
0928 08:34:32.091837 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:32.091882 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 08:34:32.093643 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:34:32.095115 (+ 1472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:34:32.095127 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:34:32.095884 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:34:32.096716 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:32.096759 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:32.098334 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:32.098342 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:32.100432 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:32.100451 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:32.101484 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:32.101492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:32.101819 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:32.102557 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:34:32.102591 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:34:32.105659 (+ 3068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:34:32.110886 (+ 5227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:34:32.110893 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:34:32.110904 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:34:32.111338 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:34:32.111897 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:34:32.111900 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:34:32.111902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:34:32.112071 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:34:32.113162 (+ 1091us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:34:32.113169 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:34:32.113470 (+ 301us) client_negotiation.cc:769] Sending connection context
0928 08:34:32.113696 (+ 226us) client_negotiation.cc:240] Negotiation successful
0928 08:34:32.114131 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":363,"threads_started":1}
I20260928 08:34:32.115850 32142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:32.090693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:57629 (local address 127.15.134.254:46653)
0928 08:34:32.091319 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:32.091330 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:32.092001 (+ 671us) server_negotiation.cc:407] Connection header received
0928 08:34:32.094212 (+ 2211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:32.094225 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:32.094587 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:32.094892 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:34:32.096963 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:32.098162 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:32.100575 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:32.101250 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:32.101386 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:32.105972 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:34:32.105986 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:34:32.106014 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:34:32.106078 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:34:32.110716 (+ 4638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:32.111507 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:32.111519 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:32.111530 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:32.111773 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:32.112239 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:32.112247 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:32.112253 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:32.113016 (+ 763us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:34:32.113269 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:32.114019 (+ 750us) server_negotiation.cc:299] Negotiation successful
0928 08:34:32.114498 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":280,"threads_started":1}
I20260928 08:34:32.118880 32140 heartbeater.cc:347] Connected to a master server at 127.15.134.254:46653
I20260928 08:34:32.119217 32140 heartbeater.cc:464] Registering TS with master...
I20260928 08:34:32.120028 32140 heartbeater.cc:511] Master 127.15.134.254:46653 requested a full tablet report, sending...
I20260928 08:34:32.122699 31955 ts_manager.cc:194] Registered new tserver with Master: ee2bdf94afca4a05bc4a120205a1bb6e (127.15.134.193:43493)
I20260928 08:34:32.124468 31955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.193@KRBTEST.COM'} at 127.15.134.193:57629
W20260928 08:34:32.447090 32144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:34:32.447428 32144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:34:32.447599 32144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:34:32.455318 32144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:34:32.455475 32144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:34:32.455662 32144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:34:32.465744 32144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:34745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:44873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=43883
--tserver_master_addrs=127.15.134.254:46653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:34:32.467036 32144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:34:32.468405 32144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:34:32.482650 32152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:34:32.482684 32150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:34:32.482615 32149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:34:32.483487 32144 server_base.cc:1034] running on GCE node
I20260928 08:34:32.484076 32144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:34:32.485112 32144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:34:32.486356 32144 hybrid_clock.cc:648] HybridClock initialized: now 1790584472486266 us; error 44 us; skew 500 ppm
I20260928 08:34:32.486702 32144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:34:32 dist-test-slave-qp2n krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584472, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:34:32.490031 32144 init.cc:377] Logged in from keytab as kudu/127.15.134.194@KRBTEST.COM (short username kudu)
I20260928 08:34:32.491779 32144 webserver.cc:460] Webserver started at http://127.15.134.194:43883/ using document root <none> and password file <none>
I20260928 08:34:32.492543 32144 fs_manager.cc:359] Metadata directory not provided
I20260928 08:34:32.492771 32144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:34:32.498569 32144 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 08:34:32.501502 32159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:32.503014 32144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 08:34:32.503233 32144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "418589322bf34fd9a0f83d7516d32051"
format_stamp: "Formatted at 2026-09-28 08:34:09 on dist-test-slave-qp2n"
server_key: "d6a8ffaf1c0d3f933b195b2a496bd25e"
server_key_iv: "cd0ad2cf6db0b35a77953ca261cef488"
server_key_version: "encryptionkey@0"
I20260928 08:34:32.504333 32144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:34:32.527773 32144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:34:32.532039 32144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:34:32.532367 32144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:34:32.533432 32144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:34:32.534826 32144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:34:32.534955 32144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:32.535095 32144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:34:32.535187 32144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:32.570811 32144 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:44873
I20260928 08:34:32.570863 32271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:44873 every 8 connection(s)
I20260928 08:34:32.572309 32144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:34:32.576854 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 32144
I20260928 08:34:32.577659 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:35431
--local_ip_for_outbound_sockets=127.15.134.195
--tserver_master_addrs=127.15.134.254:46653
--webserver_port=45465
--webserver_interface=127.15.134.195
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:34745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:34:32 dist-test-slave-qp2n krb5kdc[31369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584472, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:34:32.598227 32142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:32.575314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:40789 (local address 127.15.134.254:46653)
0928 08:34:32.575511 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:32.575517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:32.576966 (+ 1449us) server_negotiation.cc:407] Connection header received
0928 08:34:32.580067 (+ 3101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:32.580072 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:32.580175 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:32.580360 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:34:32.582776 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:32.583633 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:32.586293 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:32.586933 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:32.587085 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:32.590921 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:34:32.590930 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:34:32.590938 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:34:32.590987 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:34:32.594642 (+ 3655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:32.595433 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:32.595441 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:32.595447 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:32.595612 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:32.596036 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:32.596044 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:32.596050 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:32.596461 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:34:32.596646 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:32.597476 (+ 830us) server_negotiation.cc:299] Negotiation successful
0928 08:34:32.597744 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:34:32.598940 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:32.575798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.194:40789)
0928 08:34:32.576736 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:32.576803 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 08:34:32.579681 (+ 2878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:34:32.580715 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:34:32.580731 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:34:32.581623 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:34:32.582531 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:32.582562 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:32.583824 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:32.583832 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:32.586098 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:32.586118 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:32.587176 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:32.587182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:32.587477 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:32.588218 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:34:32.588315 (+ 97us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:34:32.590645 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:34:32.594814 (+ 4169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:34:32.594821 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:34:32.594838 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:34:32.595286 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:34:32.595715 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:34:32.595719 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:34:32.595721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:34:32.595907 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:34:32.596578 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:34:32.596584 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:34:32.597089 (+ 505us) client_negotiation.cc:769] Sending connection context
0928 08:34:32.597321 (+ 232us) client_negotiation.cc:240] Negotiation successful
0928 08:34:32.597716 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":310,"threads_started":1}
I20260928 08:34:32.600703 32272 heartbeater.cc:347] Connected to a master server at 127.15.134.254:46653
I20260928 08:34:32.601054 32272 heartbeater.cc:464] Registering TS with master...
I20260928 08:34:32.601789 32272 heartbeater.cc:511] Master 127.15.134.254:46653 requested a full tablet report, sending...
I20260928 08:34:32.603381 31955 ts_manager.cc:194] Registered new tserver with Master: 418589322bf34fd9a0f83d7516d32051 (127.15.134.194:44873)
I20260928 08:34:32.604180 31955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.194@KRBTEST.COM'} at 127.15.134.194:40789
W20260928 08:34:32.928998 32275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:34:32.929299 32275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:34:32.929479 32275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:34:32.936815 32275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:34:32.936931 32275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:34:32.937151 32275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:34:32.947041 32275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:34745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:35431
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=45465
--tserver_master_addrs=127.15.134.254:46653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:34:32.948398 32275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:34:32.949795 32275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:34:32.963790 32280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:34:32.963706 32283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:34:32.963846 32281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:34:32.964834 32275 server_base.cc:1034] running on GCE node
I20260928 08:34:32.965559 32275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:34:32.966588 32275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:34:32.967839 32275 hybrid_clock.cc:648] HybridClock initialized: now 1790584472967775 us; error 53 us; skew 500 ppm
I20260928 08:34:32.968179 32275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:34:32 dist-test-slave-qp2n krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584472, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:34:32.971501 32275 init.cc:377] Logged in from keytab as kudu/127.15.134.195@KRBTEST.COM (short username kudu)
I20260928 08:34:32.973233 32275 webserver.cc:460] Webserver started at http://127.15.134.195:45465/ using document root <none> and password file <none>
I20260928 08:34:32.973999 32275 fs_manager.cc:359] Metadata directory not provided
I20260928 08:34:32.974176 32275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:34:32.979777 32275 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 08:34:32.982738 32290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:32.984165 32275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:34:32.984393 32275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "a16c6b5ccc094350af081af313e102f3"
format_stamp: "Formatted at 2026-09-28 08:34:10 on dist-test-slave-qp2n"
server_key: "a2437f1368815a68053c4fb84734a1f3"
server_key_iv: "feb16b090e18bb2e3bb3e4a43ee0c910"
server_key_version: "encryptionkey@0"
I20260928 08:34:32.985486 32275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:34:33.019970 32275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:34:33.024493 32275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:34:33.024808 32275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:34:33.025904 32275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:34:33.027359 32275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:34:33.027482 32275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:33.027626 32275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:34:33.027716 32275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:33.062652 32275 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:35431
I20260928 08:34:33.062745 32402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:35431 every 8 connection(s)
I20260928 08:34:33.064142 32275 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:34:33.068960 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 32275
Sep 28 08:34:33 dist-test-slave-qp2n krb5kdc[31369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584472, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:34:33.088297 32407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:33.070009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42600 (local address 127.15.134.254:46653)
0928 08:34:33.070598 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:33.070604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:33.071123 (+ 519us) server_negotiation.cc:407] Connection header received
0928 08:34:33.071308 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:33.071312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:33.071371 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:33.071468 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:34:33.073564 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.074694 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.076751 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.077584 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.077869 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:33.081294 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:34:33.081299 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:34:33.081303 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:34:33.081336 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:34:33.085090 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:33.085962 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:33.085967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:33.085969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:33.086134 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:33.086600 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:33.086605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:33.086607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:33.087023 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:34:33.087305 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:33.087636 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 08:34:33.087877 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":477,"spinlock_wait_cycles":34816,"thread_start_us":336,"threads_started":1}
I20260928 08:34:33.089170 32142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:33.066912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:35315 (local address 127.15.134.254:46653)
0928 08:34:33.067130 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:33.067135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:33.068465 (+ 1330us) server_negotiation.cc:407] Connection header received
0928 08:34:33.070483 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:33.070489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:33.070573 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:33.070656 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:34:33.073267 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.074461 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.076868 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.077622 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.077750 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:33.081904 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:34:33.081911 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:34:33.081917 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:34:33.081967 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:34:33.085560 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:33.086428 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:33.086432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:33.086435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:33.086569 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:33.087068 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:33.087073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:33.087077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:33.087438 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:34:33.087569 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:33.088493 (+ 924us) server_negotiation.cc:299] Negotiation successful
0928 08:34:33.088783 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":75,"spinlock_wait_cycles":2304}
I20260928 08:34:33.090402 32405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:33.067392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.195:35315)
0928 08:34:33.068220 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:33.068274 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 08:34:33.070117 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:34:33.071232 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:34:33.071245 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:34:33.072108 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:34:33.073017 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:33.073047 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.074640 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.074644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:33.076734 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:33.076754 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.077889 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.077897 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:33.078236 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:33.078848 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:34:33.078884 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:34:33.081676 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:34:33.085719 (+ 4043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:34:33.085731 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:34:33.085751 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:34:33.086303 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:34:33.086692 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:34:33.086699 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:34:33.086705 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:34:33.086929 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:34:33.087635 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:34:33.087643 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:34:33.088061 (+ 418us) client_negotiation.cc:769] Sending connection context
0928 08:34:33.088364 (+ 303us) client_negotiation.cc:240] Negotiation successful
0928 08:34:33.088951 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":365,"threads_started":1}
I20260928 08:34:33.091806 32403 heartbeater.cc:347] Connected to a master server at 127.15.134.254:46653
I20260928 08:34:33.092121 32403 heartbeater.cc:464] Registering TS with master...
I20260928 08:34:33.092980 32403 heartbeater.cc:511] Master 127.15.134.254:46653 requested a full tablet report, sending...
I20260928 08:34:33.094421 31955 ts_manager.cc:194] Registered new tserver with Master: a16c6b5ccc094350af081af313e102f3 (127.15.134.195:35431)
I20260928 08:34:33.095284 31955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.195@KRBTEST.COM'} at 127.15.134.195:35315
I20260928 08:34:33.104355 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:34:33.111671 32407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:33.105302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42614 (local address 127.15.134.254:46653)
0928 08:34:33.105506 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:33.105515 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:33.106053 (+ 538us) server_negotiation.cc:407] Connection header received
0928 08:34:33.106249 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:33.106252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:33.106420 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:33.106587 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:34:33.107152 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.108168 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.109867 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.110315 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.110409 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:33.110866 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:34:33.111069 (+ 203us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 08:34:33.111381 (+ 312us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42614: token expired
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 08:34:33.111810 32407 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42614: token expired
W20260928 08:34:33.112270 31878 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.15.134.254:46653)
I20260928 08:34:33.125766 32407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:33.112732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42624 (local address 127.15.134.254:46653)
0928 08:34:33.112927 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:33.112931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:33.112974 (+ 43us) server_negotiation.cc:407] Connection header received
0928 08:34:33.113106 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:33.113110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:33.113163 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:33.113236 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:34:33.114571 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.115346 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.116693 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.117117 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.117323 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:33.119961 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:34:33.119965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:34:33.119968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:34:33.120001 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:34:33.122864 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:33.123597 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:33.123601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:33.123603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:33.123744 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:33.124071 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:33.124074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:33.124076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:33.124534 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:34:33.124790 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:33.125116 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 08:34:33.125434 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 08:34:33.128862 32140 heartbeater.cc:503] Master 127.15.134.254:46653 was elected leader, sending a full tablet report...
I20260928 08:34:33.129554 31955 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:34:33.137540 31955 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:34:33.161444 32413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:33.149838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:44873 (local address 127.0.0.1:44706)
0928 08:34:33.152994 (+ 3156us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:33.153044 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 08:34:33.153225 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:34:33.154242 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:34:33.154248 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:34:33.154275 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:34:33.154928 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:33.154942 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.156899 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.156904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:33.158962 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:33.158977 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.160637 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.160644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:33.160847 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:33.160885 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 08:34:33.160988 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 08:34:33.161120 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2884,"mutex_wait_us":50,"spinlock_wait_cycles":18048,"thread_start_us":520,"threads_started":1}
I20260928 08:34:33.161527 32414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:33.150016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44706 (local address 127.15.134.194:44873)
0928 08:34:33.153148 (+ 3132us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:33.153161 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:33.153197 (+ 36us) server_negotiation.cc:407] Connection header received
0928 08:34:33.153519 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:33.153531 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:33.153969 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:33.154704 (+ 735us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:34:33.155141 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.156753 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.159112 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.160335 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.160554 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:33.160764 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:33.161057 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 08:34:33.161237 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2919,"thread_start_us":2814,"threads_started":1}
I20260928 08:34:33.162762 32415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:33.150265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:43493 (local address 127.0.0.1:52522)
0928 08:34:33.153505 (+ 3240us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:33.153522 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:34:33.153635 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:34:33.154354 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:34:33.154362 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:34:33.154422 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:34:33.154846 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:33.154863 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.156501 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.156507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:33.158315 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:33.158328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.161950 (+ 3622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.161956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:33.162084 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:33.162106 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 08:34:33.162215 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 08:34:33.162350 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3124,"mutex_wait_us":6,"spinlock_wait_cycles":9216,"thread_start_us":2072,"threads_started":1}
I20260928 08:34:33.162771 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:33.150808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52522 (local address 127.15.134.193:43493)
0928 08:34:33.152262 (+ 1454us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:33.153772 (+ 1510us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:33.153802 (+ 30us) server_negotiation.cc:407] Connection header received
0928 08:34:33.153932 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:33.153941 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:33.154161 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:33.154368 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:34:33.155053 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.156329 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.158471 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.160031 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.160233 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:33.160412 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:33.162286 (+ 1874us) server_negotiation.cc:299] Negotiation successful
0928 08:34:33.162448 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1059,"thread_start_us":902,"threads_started":1}
I20260928 08:34:33.163496 32418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:33.152678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:35431 (local address 127.0.0.1:43808)
0928 08:34:33.154258 (+ 1580us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:33.154276 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:34:33.154510 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:34:33.155389 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:34:33.155393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:34:33.155418 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:34:33.155763 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:33.155774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.158584 (+ 2810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.158590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:33.160663 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:33.160675 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.162677 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.162684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:33.162878 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:33.162918 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 08:34:33.163005 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 08:34:33.163116 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1426,"mutex_wait_us":850,"spinlock_wait_cycles":231552,"thread_start_us":432,"threads_started":1}
I20260928 08:34:33.163832 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:33.151109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43808 (local address 127.15.134.195:35431)
0928 08:34:33.153488 (+ 2379us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:33.153496 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:33.154426 (+ 930us) server_negotiation.cc:407] Connection header received
0928 08:34:33.154731 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:33.154740 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:33.154955 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:33.155114 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:34:33.156009 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.158446 (+ 2437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.160828 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.162176 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.162360 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:33.162509 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:33.163212 (+ 703us) server_negotiation.cc:299] Negotiation successful
0928 08:34:33.163375 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2209,"thread_start_us":2054,"threads_started":1}
I20260928 08:34:33.165463 32207 tablet_service.cc:1467] Processing CreateTablet for tablet 35cfe713b3ee41789c80eba89727d627 (DEFAULT_TABLE table=test-table [id=58a6770e9aaa4175ba556b781206f224]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:34:33.166669 32207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35cfe713b3ee41789c80eba89727d627. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:34:33.166427 32075 tablet_service.cc:1467] Processing CreateTablet for tablet 35cfe713b3ee41789c80eba89727d627 (DEFAULT_TABLE table=test-table [id=58a6770e9aaa4175ba556b781206f224]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:34:33.167892 32075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35cfe713b3ee41789c80eba89727d627. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:34:33.168646 32338 tablet_service.cc:1467] Processing CreateTablet for tablet 35cfe713b3ee41789c80eba89727d627 (DEFAULT_TABLE table=test-table [id=58a6770e9aaa4175ba556b781206f224]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:34:33.170022 32338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35cfe713b3ee41789c80eba89727d627. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:34:33.178937 32419 tablet_bootstrap.cc:492] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051: Bootstrap starting.
I20260928 08:34:33.182255 32419 tablet_bootstrap.cc:654] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051: Neither blocks nor log segments found. Creating new log.
I20260928 08:34:33.183698 32419 log.cc:824] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051: Log is configured to *not* fsync() on all Append() calls
I20260928 08:34:33.183825 32420 tablet_bootstrap.cc:492] T 35cfe713b3ee41789c80eba89727d627 P ee2bdf94afca4a05bc4a120205a1bb6e: Bootstrap starting.
I20260928 08:34:33.185415 32422 tablet_bootstrap.cc:492] T 35cfe713b3ee41789c80eba89727d627 P a16c6b5ccc094350af081af313e102f3: Bootstrap starting.
I20260928 08:34:33.187669 32420 tablet_bootstrap.cc:654] T 35cfe713b3ee41789c80eba89727d627 P ee2bdf94afca4a05bc4a120205a1bb6e: Neither blocks nor log segments found. Creating new log.
I20260928 08:34:33.188449 32422 tablet_bootstrap.cc:654] T 35cfe713b3ee41789c80eba89727d627 P a16c6b5ccc094350af081af313e102f3: Neither blocks nor log segments found. Creating new log.
I20260928 08:34:33.189018 32419 tablet_bootstrap.cc:492] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051: No bootstrap required, opened a new log
I20260928 08:34:33.188983 32420 log.cc:824] T 35cfe713b3ee41789c80eba89727d627 P ee2bdf94afca4a05bc4a120205a1bb6e: Log is configured to *not* fsync() on all Append() calls
I20260928 08:34:33.189286 32419 ts_tablet_manager.cc:1397] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260928 08:34:33.189733 32422 log.cc:824] T 35cfe713b3ee41789c80eba89727d627 P a16c6b5ccc094350af081af313e102f3: Log is configured to *not* fsync() on all Append() calls
I20260928 08:34:33.192469 32420 tablet_bootstrap.cc:492] T 35cfe713b3ee41789c80eba89727d627 P ee2bdf94afca4a05bc4a120205a1bb6e: No bootstrap required, opened a new log
I20260928 08:34:33.192763 32420 ts_tablet_manager.cc:1397] T 35cfe713b3ee41789c80eba89727d627 P ee2bdf94afca4a05bc4a120205a1bb6e: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260928 08:34:33.192919 32422 tablet_bootstrap.cc:492] T 35cfe713b3ee41789c80eba89727d627 P a16c6b5ccc094350af081af313e102f3: No bootstrap required, opened a new log
I20260928 08:34:33.193166 32422 ts_tablet_manager.cc:1397] T 35cfe713b3ee41789c80eba89727d627 P a16c6b5ccc094350af081af313e102f3: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 08:34:33.199803 32420 raft_consensus.cc:348] T 35cfe713b3ee41789c80eba89727d627 P ee2bdf94afca4a05bc4a120205a1bb6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418589322bf34fd9a0f83d7516d32051" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44873 } } peers { permanent_uuid: "ee2bdf94afca4a05bc4a120205a1bb6e" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 43493 } } peers { permanent_uuid: "a16c6b5ccc094350af081af313e102f3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 35431 } }
I20260928 08:34:33.200233 32420 raft_consensus.cc:374] T 35cfe713b3ee41789c80eba89727d627 P ee2bdf94afca4a05bc4a120205a1bb6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:34:33.200059 32419 raft_consensus.cc:348] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418589322bf34fd9a0f83d7516d32051" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44873 } } peers { permanent_uuid: "ee2bdf94afca4a05bc4a120205a1bb6e" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 43493 } } peers { permanent_uuid: "a16c6b5ccc094350af081af313e102f3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 35431 } }
I20260928 08:34:33.200356 32420 raft_consensus.cc:729] T 35cfe713b3ee41789c80eba89727d627 P ee2bdf94afca4a05bc4a120205a1bb6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee2bdf94afca4a05bc4a120205a1bb6e, State: Initialized, Role: FOLLOWER
I20260928 08:34:33.200492 32419 raft_consensus.cc:374] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:34:33.200595 32419 raft_consensus.cc:729] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 418589322bf34fd9a0f83d7516d32051, State: Initialized, Role: FOLLOWER
I20260928 08:34:33.201073 32420 consensus_queue.cc:260] T 35cfe713b3ee41789c80eba89727d627 P ee2bdf94afca4a05bc4a120205a1bb6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418589322bf34fd9a0f83d7516d32051" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44873 } } peers { permanent_uuid: "ee2bdf94afca4a05bc4a120205a1bb6e" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 43493 } } peers { permanent_uuid: "a16c6b5ccc094350af081af313e102f3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 35431 } }
I20260928 08:34:33.201267 32419 consensus_queue.cc:260] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418589322bf34fd9a0f83d7516d32051" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44873 } } peers { permanent_uuid: "ee2bdf94afca4a05bc4a120205a1bb6e" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 43493 } } peers { permanent_uuid: "a16c6b5ccc094350af081af313e102f3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 35431 } }
I20260928 08:34:33.204864 32420 ts_tablet_manager.cc:1428] T 35cfe713b3ee41789c80eba89727d627 P ee2bdf94afca4a05bc4a120205a1bb6e: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260928 08:34:33.204564 32422 raft_consensus.cc:348] T 35cfe713b3ee41789c80eba89727d627 P a16c6b5ccc094350af081af313e102f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418589322bf34fd9a0f83d7516d32051" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44873 } } peers { permanent_uuid: "ee2bdf94afca4a05bc4a120205a1bb6e" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 43493 } } peers { permanent_uuid: "a16c6b5ccc094350af081af313e102f3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 35431 } }
I20260928 08:34:33.205176 32422 raft_consensus.cc:374] T 35cfe713b3ee41789c80eba89727d627 P a16c6b5ccc094350af081af313e102f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:34:33.205435 32422 raft_consensus.cc:729] T 35cfe713b3ee41789c80eba89727d627 P a16c6b5ccc094350af081af313e102f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a16c6b5ccc094350af081af313e102f3, State: Initialized, Role: FOLLOWER
I20260928 08:34:33.206192 32422 consensus_queue.cc:260] T 35cfe713b3ee41789c80eba89727d627 P a16c6b5ccc094350af081af313e102f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418589322bf34fd9a0f83d7516d32051" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44873 } } peers { permanent_uuid: "ee2bdf94afca4a05bc4a120205a1bb6e" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 43493 } } peers { permanent_uuid: "a16c6b5ccc094350af081af313e102f3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 35431 } }
I20260928 08:34:33.207463 32403 heartbeater.cc:503] Master 127.15.134.254:46653 was elected leader, sending a full tablet report...
I20260928 08:34:33.208271 32422 ts_tablet_manager.cc:1428] T 35cfe713b3ee41789c80eba89727d627 P a16c6b5ccc094350af081af313e102f3: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260928 08:34:33.209367 32272 heartbeater.cc:503] Master 127.15.134.254:46653 was elected leader, sending a full tablet report...
I20260928 08:34:33.210657 32419 ts_tablet_manager.cc:1428] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051: Time spent starting tablet: real 0.021s user 0.012s sys 0.008s
I20260928 08:34:33.231969 32426 raft_consensus.cc:482] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:34:33.232475 32426 raft_consensus.cc:504] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418589322bf34fd9a0f83d7516d32051" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44873 } } peers { permanent_uuid: "ee2bdf94afca4a05bc4a120205a1bb6e" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 43493 } } peers { permanent_uuid: "a16c6b5ccc094350af081af313e102f3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 35431 } }
I20260928 08:34:33.234560 32426 leader_election.cc:288] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee2bdf94afca4a05bc4a120205a1bb6e (127.15.134.193:43493), a16c6b5ccc094350af081af313e102f3 (127.15.134.195:35431)
I20260928 08:34:33.244305 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:33.235189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:36321 (local address 127.15.134.195:35431)
0928 08:34:33.235518 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:33.235529 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:33.236384 (+ 855us) server_negotiation.cc:407] Connection header received
0928 08:34:33.236497 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:33.236502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:33.236576 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:33.236697 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:34:33.237445 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.238628 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.241771 (+ 3143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.243113 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.243284 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:33.243364 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:33.243890 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 08:34:33.244030 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260928 08:34:33.244305 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:33.235189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:59659 (local address 127.15.134.193:43493)
0928 08:34:33.235744 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:33.235753 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:33.236006 (+ 253us) server_negotiation.cc:407] Connection header received
0928 08:34:33.236265 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:33.236270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:33.236387 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:33.236498 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:34:33.237377 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.238563 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.241813 (+ 3250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.243094 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.243284 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:33.243364 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:33.243890 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 08:34:33.244032 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375}
I20260928 08:34:33.244524 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:33.234841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:43493 (local address 127.15.134.194:59659)
0928 08:34:33.235855 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:33.235881 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 08:34:33.236060 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:34:33.236546 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:34:33.236549 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:34:33.236568 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:34:33.237093 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:33.237112 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.238791 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.238798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:33.241567 (+ 2769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:33.241586 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.243406 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.243413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:33.243687 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:33.243737 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 08:34:33.243841 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 08:34:33.244043 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":230,"thread_start_us":602,"threads_started":1}
I20260928 08:34:33.244604 32429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:33.235069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:35431 (local address 127.15.134.194:36321)
0928 08:34:33.236164 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:33.236180 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:34:33.236390 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:34:33.236718 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:34:33.236724 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:34:33.236766 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:34:33.237258 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:33.237272 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.238791 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.238798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:33.241566 (+ 2768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:34:33.241586 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.243407 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.243413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:34:33.243682 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:33.243734 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 08:34:33.243841 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 08:34:33.244043 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"mutex_wait_us":241,"thread_start_us":722,"threads_started":1}
I20260928 08:34:33.245939 32358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cfe713b3ee41789c80eba89727d627" candidate_uuid: "418589322bf34fd9a0f83d7516d32051" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a16c6b5ccc094350af081af313e102f3" is_pre_election: true
I20260928 08:34:33.245937 32095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cfe713b3ee41789c80eba89727d627" candidate_uuid: "418589322bf34fd9a0f83d7516d32051" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee2bdf94afca4a05bc4a120205a1bb6e" is_pre_election: true
I20260928 08:34:33.246533 32358 raft_consensus.cc:2454] T 35cfe713b3ee41789c80eba89727d627 P a16c6b5ccc094350af081af313e102f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 418589322bf34fd9a0f83d7516d32051 in term 0.
I20260928 08:34:33.246527 32095 raft_consensus.cc:2454] T 35cfe713b3ee41789c80eba89727d627 P ee2bdf94afca4a05bc4a120205a1bb6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 418589322bf34fd9a0f83d7516d32051 in term 0.
I20260928 08:34:33.247267 32162 leader_election.cc:302] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 418589322bf34fd9a0f83d7516d32051, ee2bdf94afca4a05bc4a120205a1bb6e; no voters:
I20260928 08:34:33.247705 32426 raft_consensus.cc:2781] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:34:33.247864 32426 raft_consensus.cc:482] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:34:33.247977 32426 raft_consensus.cc:3037] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:34:33.250340 32426 raft_consensus.cc:504] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418589322bf34fd9a0f83d7516d32051" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44873 } } peers { permanent_uuid: "ee2bdf94afca4a05bc4a120205a1bb6e" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 43493 } } peers { permanent_uuid: "a16c6b5ccc094350af081af313e102f3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 35431 } }
I20260928 08:34:33.250999 32426 leader_election.cc:288] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051 [CANDIDATE]: Term 1 election: Requested vote from peers ee2bdf94afca4a05bc4a120205a1bb6e (127.15.134.193:43493), a16c6b5ccc094350af081af313e102f3 (127.15.134.195:35431)
I20260928 08:34:33.251436 32095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cfe713b3ee41789c80eba89727d627" candidate_uuid: "418589322bf34fd9a0f83d7516d32051" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee2bdf94afca4a05bc4a120205a1bb6e"
I20260928 08:34:33.251645 32095 raft_consensus.cc:3037] T 35cfe713b3ee41789c80eba89727d627 P ee2bdf94afca4a05bc4a120205a1bb6e [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:34:33.251633 32358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35cfe713b3ee41789c80eba89727d627" candidate_uuid: "418589322bf34fd9a0f83d7516d32051" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a16c6b5ccc094350af081af313e102f3"
I20260928 08:34:33.251837 32358 raft_consensus.cc:3037] T 35cfe713b3ee41789c80eba89727d627 P a16c6b5ccc094350af081af313e102f3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:34:33.254058 32095 raft_consensus.cc:2454] T 35cfe713b3ee41789c80eba89727d627 P ee2bdf94afca4a05bc4a120205a1bb6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 418589322bf34fd9a0f83d7516d32051 in term 1.
I20260928 08:34:33.254169 32358 raft_consensus.cc:2454] T 35cfe713b3ee41789c80eba89727d627 P a16c6b5ccc094350af081af313e102f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 418589322bf34fd9a0f83d7516d32051 in term 1.
I20260928 08:34:33.254521 32162 leader_election.cc:302] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 418589322bf34fd9a0f83d7516d32051, ee2bdf94afca4a05bc4a120205a1bb6e; no voters:
I20260928 08:34:33.254827 32426 raft_consensus.cc:2781] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:34:33.255333 32426 raft_consensus.cc:686] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051 [term 1 LEADER]: Becoming Leader. State: Replica: 418589322bf34fd9a0f83d7516d32051, State: Running, Role: LEADER
I20260928 08:34:33.255874 32426 consensus_queue.cc:237] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418589322bf34fd9a0f83d7516d32051" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44873 } } peers { permanent_uuid: "ee2bdf94afca4a05bc4a120205a1bb6e" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 43493 } } peers { permanent_uuid: "a16c6b5ccc094350af081af313e102f3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 35431 } }
I20260928 08:34:33.260394 31954 catalog_manager.cc:5495] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051 reported cstate change: term changed from 0 to 1, leader changed from <none> to 418589322bf34fd9a0f83d7516d32051 (127.15.134.194). New cstate: current_term: 1 leader_uuid: "418589322bf34fd9a0f83d7516d32051" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "418589322bf34fd9a0f83d7516d32051" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee2bdf94afca4a05bc4a120205a1bb6e" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 43493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a16c6b5ccc094350af081af313e102f3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 35431 } health_report { overall_health: UNKNOWN } } }
I20260928 08:34:33.288842 32414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:33.283118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44712 (local address 127.15.134.194:44873)
0928 08:34:33.283324 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:33.283331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:33.283452 (+ 121us) server_negotiation.cc:407] Connection header received
0928 08:34:33.283697 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:33.283702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:33.283836 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:33.283996 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:34:33.284357 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.285152 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.286634 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.287056 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.287189 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:33.287682 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:34:33.288181 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:34:33.288341 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:33.288423 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 08:34:33.288506 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 08:34:33.297297 32358 raft_consensus.cc:1261] T 35cfe713b3ee41789c80eba89727d627 P a16c6b5ccc094350af081af313e102f3 [term 1 FOLLOWER]: Refusing update from remote peer 418589322bf34fd9a0f83d7516d32051: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:34:33.297307 32095 raft_consensus.cc:1261] T 35cfe713b3ee41789c80eba89727d627 P ee2bdf94afca4a05bc4a120205a1bb6e [term 1 FOLLOWER]: Refusing update from remote peer 418589322bf34fd9a0f83d7516d32051: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:34:33.298214 32430 consensus_queue.cc:1035] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee2bdf94afca4a05bc4a120205a1bb6e" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 43493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:34:33.298554 32426 consensus_queue.cc:1035] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a16c6b5ccc094350af081af313e102f3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 35431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:34:33.308682 32432 mvcc.cc:205] Tried to move back new op lower bound from 7334234002614685696 to 7334234002460254208. Current Snapshot: MvccSnapshot[applied={T|T < 7334234002614685696}]
I20260928 08:34:33.315341 32434 mvcc.cc:205] Tried to move back new op lower bound from 7334234002614685696 to 7334234002460254208. Current Snapshot: MvccSnapshot[applied={T|T < 7334234002614685696}]
I20260928 08:34:33.322372 31955 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:34:33.322682 31955 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:34:33.326915 31955 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b: Sending DeleteTablet for 3 replicas of tablet 35cfe713b3ee41789c80eba89727d627
I20260928 08:34:33.328235 32075 tablet_service.cc:1514] Processing DeleteTablet for tablet 35cfe713b3ee41789c80eba89727d627 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:34:33 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:52522
I20260928 08:34:33.328208 32207 tablet_service.cc:1514] Processing DeleteTablet for tablet 35cfe713b3ee41789c80eba89727d627 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:34:33 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:44706
I20260928 08:34:33.328352 32338 tablet_service.cc:1514] Processing DeleteTablet for tablet 35cfe713b3ee41789c80eba89727d627 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:34:33 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:43808
I20260928 08:34:33.331430 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 31992
I20260928 08:34:33.342695 32442 tablet_replica.cc:330] T 35cfe713b3ee41789c80eba89727d627 P a16c6b5ccc094350af081af313e102f3: stopping tablet replica
I20260928 08:34:33.343278 32442 raft_consensus.cc:2229] T 35cfe713b3ee41789c80eba89727d627 P a16c6b5ccc094350af081af313e102f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:34:33.343937 32442 raft_consensus.cc:2258] T 35cfe713b3ee41789c80eba89727d627 P a16c6b5ccc094350af081af313e102f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:34:33.346804 32442 ts_tablet_manager.cc:1905] T 35cfe713b3ee41789c80eba89727d627 P a16c6b5ccc094350af081af313e102f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:34:33.350723 32443 tablet_replica.cc:330] T 35cfe713b3ee41789c80eba89727d627 P ee2bdf94afca4a05bc4a120205a1bb6e: stopping tablet replica
I20260928 08:34:33.351303 32443 raft_consensus.cc:2229] T 35cfe713b3ee41789c80eba89727d627 P ee2bdf94afca4a05bc4a120205a1bb6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:34:33.351928 32443 raft_consensus.cc:2258] T 35cfe713b3ee41789c80eba89727d627 P ee2bdf94afca4a05bc4a120205a1bb6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:34:33.354388 32443 ts_tablet_manager.cc:1905] T 35cfe713b3ee41789c80eba89727d627 P ee2bdf94afca4a05bc4a120205a1bb6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:34:33.358371 32442 ts_tablet_manager.cc:1918] T 35cfe713b3ee41789c80eba89727d627 P a16c6b5ccc094350af081af313e102f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:34:33.358451 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:33.333106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52526 (local address 127.15.134.193:43493)
0928 08:34:33.333355 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:33.333365 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:33.333395 (+ 30us) server_negotiation.cc:407] Connection header received
0928 08:34:33.333482 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:33.333491 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:33.333605 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:33.333783 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:34:33.336541 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.338317 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.341056 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.341940 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.342126 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:33.346558 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:34:33.346578 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:34:33.346595 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:34:33.346661 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:34:33.351409 (+ 4748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:33.353367 (+ 1958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:33.353381 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:33.353387 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:33.353578 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:33.355128 (+ 1550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:33.355134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:33.355137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:33.355710 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:34:33.356052 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:33.357711 (+ 1659us) server_negotiation.cc:299] Negotiation successful
0928 08:34:33.357963 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 08:34:33.358947 32442 log.cc:1196] T 35cfe713b3ee41789c80eba89727d627 P a16c6b5ccc094350af081af313e102f3: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/35cfe713b3ee41789c80eba89727d627
I20260928 08:34:33.359707 32442 ts_tablet_manager.cc:1939] T 35cfe713b3ee41789c80eba89727d627 P a16c6b5ccc094350af081af313e102f3: Deleting consensus metadata
I20260928 08:34:33.360598 32135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:34:33.362399 31940 catalog_manager.cc:4841] TS a16c6b5ccc094350af081af313e102f3 (127.15.134.195:35431): tablet 35cfe713b3ee41789c80eba89727d627 (table test-table [id=58a6770e9aaa4175ba556b781206f224]) successfully deleted
I20260928 08:34:33.369319 32441 tablet_replica.cc:330] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051: stopping tablet replica
I20260928 08:34:33.369817 32441 raft_consensus.cc:2229] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:34:33.370544 32441 raft_consensus.cc:2258] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:34:33.372505 32441 ts_tablet_manager.cc:1905] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:34:33.378513 32441 ts_tablet_manager.cc:1918] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:34:33.378751 32441 log.cc:1196] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/35cfe713b3ee41789c80eba89727d627
I20260928 08:34:33.379311 32441 ts_tablet_manager.cc:1939] T 35cfe713b3ee41789c80eba89727d627 P 418589322bf34fd9a0f83d7516d32051: Deleting consensus metadata
I20260928 08:34:33.381240 31939 catalog_manager.cc:4841] TS 418589322bf34fd9a0f83d7516d32051 (127.15.134.194:44873): tablet 35cfe713b3ee41789c80eba89727d627 (table test-table [id=58a6770e9aaa4175ba556b781206f224]) successfully deleted
I20260928 08:34:33.492144 32443 ts_tablet_manager.cc:1918] T 35cfe713b3ee41789c80eba89727d627 P ee2bdf94afca4a05bc4a120205a1bb6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:34:33.492424 32443 log.cc:1196] T 35cfe713b3ee41789c80eba89727d627 P ee2bdf94afca4a05bc4a120205a1bb6e: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/wal/wals/35cfe713b3ee41789c80eba89727d627
I20260928 08:34:33.492810 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 31992
I20260928 08:34:33.512490 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 32144
W20260928 08:34:33.512490 31941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.134.193:43493 (error 108)
I20260928 08:34:33.514668 32418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:33.514182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:43493 (local address 127.0.0.1:52534)
0928 08:34:33.514375 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:33.514478 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:43493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260928 08:34:33.514930 31941 catalog_manager.cc:4584] TS ee2bdf94afca4a05bc4a120205a1bb6e (127.15.134.193:43493): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 35cfe713b3ee41789c80eba89727d627: Network error: Client connection negotiation failed: client connection to 127.15.134.193:43493: connect: Connection refused (error 111)
I20260928 08:34:33.528990 32414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:33.513852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44716 (local address 127.15.134.194:44873)
0928 08:34:33.514050 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:33.514056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:33.514078 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:34:33.514148 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:33.514154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:33.514238 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:33.514349 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:34:33.515944 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.517040 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.518328 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.518772 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.518914 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:33.521699 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:34:33.521711 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:34:33.521726 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:34:33.521761 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:34:33.525510 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:33.526442 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:33.526462 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:33.526468 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:33.526668 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:33.527099 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:33.527106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:33.527110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:33.527549 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:34:33.527938 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:33.528272 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 08:34:33.528532 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 08:34:33.529866 32267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:34:33.568652 32418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:33.568139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:43493 (local address 127.0.0.1:52542)
0928 08:34:33.568344 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:33.568451 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:43493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 08:34:33.569696 32418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:33.569331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:43493 (local address 127.0.0.1:52546)
0928 08:34:33.569500 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:33.569589 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:43493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 08:34:33.630456 32418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:33.629908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:43493 (local address 127.0.0.1:52548)
0928 08:34:33.630139 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:33.630300 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:43493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 08:34:33.631469 32418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:33.631122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:43493 (local address 127.0.0.1:52552)
0928 08:34:33.631295 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:33.631362 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:43493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 08:34:33.670499 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 32144
I20260928 08:34:33.691589 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 32275
I20260928 08:34:33.705909 32417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:33.692351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43816 (local address 127.15.134.195:35431)
0928 08:34:33.692585 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 08:34:33.692590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:34:33.692695 (+ 105us) server_negotiation.cc:407] Connection header received
0928 08:34:33.692900 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:34:33.692904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:34:33.692969 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:34:33.693053 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:34:33.694529 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.695296 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.696516 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:34:33.696973 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:34:33.697069 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:34:33.699657 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:34:33.699672 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:34:33.699688 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:34:33.699734 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:34:33.702798 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:33.703617 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:33.703624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:33.703627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:33.703799 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:34:33.704209 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:34:33.704213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:34:33.704215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:34:33.704604 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:34:33.704929 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0928 08:34:33.705281 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 08:34:33.705452 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 08:34:33.706630 32398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:34:33.712146 32418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:33.711674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:43493 (local address 127.0.0.1:52558)
0928 08:34:33.711878 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:33.711979 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:43493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 08:34:33.713418 32418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:33.712974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:43493 (local address 127.0.0.1:52568)
0928 08:34:33.713209 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:33.713299 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:43493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260928 08:34:33.797389 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 32275
I20260928 08:34:33.815447 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 31922
I20260928 08:34:33.816767 31985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:34:33.931389 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 31922
I20260928 08:34:53.955593 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:46653
--webserver_interface=127.15.134.254
--webserver_port=35445
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:34745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:46653
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:34:54.318014 32449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:34:54.318406 32449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:34:54.318584 32449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:34:54.326063 32449 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 08:34:54.326210 32449 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 08:34:54.326285 32449 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 08:34:54.326366 32449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:34:54.326447 32449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:34:54.326572 32449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:34:54.326665 32449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:34:54.326745 32449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:34:54.337003 32449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:34745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:46653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:46653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=35445
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:34:54.338418 32449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:34:54.340077 32449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:34:54.353792 32454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:34:54.353737 32457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:34:54.353822 32455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:34:54.354282 32449 server_base.cc:1034] running on GCE node
I20260928 08:34:54.355310 32449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:34:54.356950 32449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:34:54.358264 32449 hybrid_clock.cc:648] HybridClock initialized: now 1790584494358188 us; error 42 us; skew 500 ppm
I20260928 08:34:54.358644 32449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:34:54 dist-test-slave-qp2n krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584494, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:34:54.362597 32449 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:34:54.364441 32449 webserver.cc:460] Webserver started at http://127.15.134.254:35445/ using document root <none> and password file <none>
I20260928 08:34:54.365383 32449 fs_manager.cc:359] Metadata directory not provided
I20260928 08:34:54.365543 32449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:34:54.372290 32449 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 08:34:54.375645 32464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:54.377147 32449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 08:34:54.377351 32449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "200c54a5f2444d2eb26bd5b75625071b"
format_stamp: "Formatted at 2026-09-28 08:34:08 on dist-test-slave-qp2n"
server_key: "4552dde4d3458b7cfbdb36fb58f2350a"
server_key_iv: "28e6e492903605f7a9dd1a8225745b45"
server_key_version: "encryptionkey@0"
I20260928 08:34:54.378353 32449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:34:54.410516 32449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:34:54.416085 32449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:34:54.416435 32449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:34:54.439849 32449 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:46653
I20260928 08:34:54.439874 32515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:46653 every 8 connection(s)
I20260928 08:34:54.441588 32449 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:34:54.449435 32516 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:34:54.452008 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 32449
I20260928 08:34:54.452946 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:43493
--local_ip_for_outbound_sockets=127.15.134.193
--tserver_master_addrs=127.15.134.254:46653
--webserver_port=35381
--webserver_interface=127.15.134.193
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:34745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:34:54.453545 32516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b: Bootstrap starting.
I20260928 08:34:54.476859 32516 log.cc:824] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:34:54.495777 32516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:34:54.496393 32516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b: Bootstrap complete.
I20260928 08:34:54.505897 32516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200c54a5f2444d2eb26bd5b75625071b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46653 } }
I20260928 08:34:54.506830 32516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 200c54a5f2444d2eb26bd5b75625071b, State: Initialized, Role: FOLLOWER
I20260928 08:34:54.507459 32516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200c54a5f2444d2eb26bd5b75625071b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46653 } }
I20260928 08:34:54.507733 32516 raft_consensus.cc:388] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:34:54.507874 32516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:34:54.508072 32516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [term 2 FOLLOWER]: Advancing to term 3
I20260928 08:34:54.512115 32516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200c54a5f2444d2eb26bd5b75625071b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46653 } }
I20260928 08:34:54.512676 32516 leader_election.cc:302] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 200c54a5f2444d2eb26bd5b75625071b; no voters:
I20260928 08:34:54.513391 32516 leader_election.cc:288] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 08:34:54.513551 32520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [term 3 FOLLOWER]: Leader election won for term 3
I20260928 08:34:54.514745 32520 raft_consensus.cc:686] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [term 3 LEADER]: Becoming Leader. State: Replica: 200c54a5f2444d2eb26bd5b75625071b, State: Running, Role: LEADER
I20260928 08:34:54.515375 32520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200c54a5f2444d2eb26bd5b75625071b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46653 } }
I20260928 08:34:54.515710 32516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:34:54.517961 32522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 200c54a5f2444d2eb26bd5b75625071b. Latest consensus state: current_term: 3 leader_uuid: "200c54a5f2444d2eb26bd5b75625071b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200c54a5f2444d2eb26bd5b75625071b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46653 } } }
I20260928 08:34:54.518430 32522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [sys.catalog]: This master's current role is: LEADER
I20260928 08:34:54.518082 32521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "200c54a5f2444d2eb26bd5b75625071b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "200c54a5f2444d2eb26bd5b75625071b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46653 } } }
I20260928 08:34:54.518872 32521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b [sys.catalog]: This master's current role is: LEADER
I20260928 08:34:54.526715 32527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:34:54.536249 32527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:34:54.537840 32527 catalog_manager.cc:1259] Loaded cluster ID: 9c6dad54183047e6bc3504556dc7e5fb
I20260928 08:34:54.537997 32527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:34:54.542295 32527 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:34:54.544431 32527 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b: Loaded TSK: 0
I20260928 08:34:54.544734 32527 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 200c54a5f2444d2eb26bd5b75625071b: Loaded TSK: 1
F20260928 08:34:54.546015 32527 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6e008529ad google::LogMessage::Fail() at ??:0
@ 0x7f6e0085588a google::LogMessage::SendToLog() at ??:0
@ 0x7f6e008524de google::LogMessage::Flush() at ??:0
@ 0x7f6e00853a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6e045b67d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f6e045a573e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f6e162811c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f6e16280783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f6e1630c2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f6e1630c0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f6e163565e9 std::function<>::operator()() at ??:0
@ 0x7f6e16284cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f6e16282c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6e1630a610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f6e1630a439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6e0349e345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6e034a191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f6e034a1799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6e0347ceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6e067566db start_thread at ??:0
@ 0x7f6dff6ec61f clone at ??:0
W20260928 08:34:54.857456 32518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:34:54.857810 32518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:34:54.858085 32518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:34:54.865778 32518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:34:54.865962 32518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:34:54.866165 32518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:34:54.876298 32518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:34745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:43493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=35381
--tserver_master_addrs=127.15.134.254:46653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:34:54.877713 32518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:34:54.879081 32518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:34:54.893213 32563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:34:54.893256 32564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:34:54.893493 32566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:34:54.895037 32518 server_base.cc:1034] running on GCE node
I20260928 08:34:54.895714 32518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:34:54.896756 32518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:34:54.898005 32518 hybrid_clock.cc:648] HybridClock initialized: now 1790584494897948 us; error 64 us; skew 500 ppm
I20260928 08:34:54.898427 32518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:34:54 dist-test-slave-qp2n krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584494, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:34:54.901906 32518 init.cc:377] Logged in from keytab as kudu/127.15.134.193@KRBTEST.COM (short username kudu)
I20260928 08:34:54.903569 32518 webserver.cc:460] Webserver started at http://127.15.134.193:35381/ using document root <none> and password file <none>
I20260928 08:34:54.904299 32518 fs_manager.cc:359] Metadata directory not provided
I20260928 08:34:54.904449 32518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:34:54.909976 32518 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 08:34:54.913058 32573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:54.914518 32518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:34:54.914711 32518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "ee2bdf94afca4a05bc4a120205a1bb6e"
format_stamp: "Formatted at 2026-09-28 08:34:09 on dist-test-slave-qp2n"
server_key: "72c8ee8b8a5128348b43fdb6d020dba8"
server_key_iv: "1a954c4ac128359b52ba7158345e4ec0"
server_key_version: "encryptionkey@0"
I20260928 08:34:54.915714 32518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:34:54.945492 32518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:34:54.949811 32518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:34:54.950115 32518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:34:54.951167 32518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:34:54.953022 32580 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260928 08:34:54.958691 32580 ts_tablet_manager.cc:1852] T 35cfe713b3ee41789c80eba89727d627 P ee2bdf94afca4a05bc4a120205a1bb6e: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 08:34:54.959707 32580 ts_tablet_manager.cc:1905] T 35cfe713b3ee41789c80eba89727d627 P ee2bdf94afca4a05bc4a120205a1bb6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:34:54.967518 32580 ts_tablet_manager.cc:1918] T 35cfe713b3ee41789c80eba89727d627 P ee2bdf94afca4a05bc4a120205a1bb6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:34:54.967809 32580 ts_tablet_manager.cc:1939] T 35cfe713b3ee41789c80eba89727d627 P ee2bdf94afca4a05bc4a120205a1bb6e: Deleting consensus metadata
I20260928 08:34:54.968601 32518 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 08:34:54.968799 32518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.000s sys 0.000s
I20260928 08:34:54.968958 32518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:34:54.969154 32518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:55.005259 32518 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:43493
I20260928 08:34:55.005353 32686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:43493 every 8 connection(s)
I20260928 08:34:55.006637 32518 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:34:55.009850 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 32518
I20260928 08:34:55.010715 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:44873
--local_ip_for_outbound_sockets=127.15.134.194
--tserver_master_addrs=127.15.134.254:46653
--webserver_port=43883
--webserver_interface=127.15.134.194
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:34745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:34:55.012142 32689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:55.009817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.193:46355)
0928 08:34:55.010831 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:55.011151 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":667,"thread_start_us":400,"threads_started":1}
W20260928 08:34:55.012939 32687 heartbeater.cc:644] Failed to heartbeat to 127.15.134.254:46653 (0 consecutive failures): Network error: Failed to ping master at 127.15.134.254:46653: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
I20260928 08:34:55.013952 32689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:55.013598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.193:35347)
0928 08:34:55.013774 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:55.013847 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 08:34:55.014964 32689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:55.014610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.193:36167)
0928 08:34:55.014765 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:55.014849 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 08:34:55.015297 32687 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:34:55.349591 32690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:34:55.349931 32690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:34:55.350109 32690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:34:55.357677 32690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:34:55.357831 32690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:34:55.358021 32690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:34:55.368201 32690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:34745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:44873
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=43883
--tserver_master_addrs=127.15.134.254:46653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:34:55.369628 32690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:34:55.370986 32690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:34:55.385905 32698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:34:55.385823 32696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:34:55.386152 32695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:34:55.386914 32690 server_base.cc:1034] running on GCE node
I20260928 08:34:55.387493 32690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:34:55.388479 32690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:34:55.389698 32690 hybrid_clock.cc:648] HybridClock initialized: now 1790584495389643 us; error 51 us; skew 500 ppm
I20260928 08:34:55.390045 32690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:34:55 dist-test-slave-qp2n krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584495, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:34:55.393370 32690 init.cc:377] Logged in from keytab as kudu/127.15.134.194@KRBTEST.COM (short username kudu)
I20260928 08:34:55.394903 32690 webserver.cc:460] Webserver started at http://127.15.134.194:43883/ using document root <none> and password file <none>
I20260928 08:34:55.395612 32690 fs_manager.cc:359] Metadata directory not provided
I20260928 08:34:55.395759 32690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:34:55.401391 32690 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 08:34:55.404167 32705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:55.405526 32690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:34:55.405722 32690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "418589322bf34fd9a0f83d7516d32051"
format_stamp: "Formatted at 2026-09-28 08:34:09 on dist-test-slave-qp2n"
server_key: "d6a8ffaf1c0d3f933b195b2a496bd25e"
server_key_iv: "cd0ad2cf6db0b35a77953ca261cef488"
server_key_version: "encryptionkey@0"
I20260928 08:34:55.406677 32690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:34:55.435585 32690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:34:55.439721 32690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:34:55.440001 32690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:34:55.441270 32690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:34:55.442801 32690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:34:55.442919 32690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:55.443029 32690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:34:55.443127 32690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:55.478236 32690 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:44873
I20260928 08:34:55.478322 349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:44873 every 8 connection(s)
I20260928 08:34:55.479743 32690 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:34:55.485399 352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:55.482859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.194:50515)
0928 08:34:55.483599 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:55.483857 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":284,"threads_started":1}
W20260928 08:34:55.486316 350 heartbeater.cc:644] Failed to heartbeat to 127.15.134.254:46653 (0 consecutive failures): Network error: Failed to ping master at 127.15.134.254:46653: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
I20260928 08:34:55.487473 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 32690
I20260928 08:34:55.487560 352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:55.487046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.194:43599)
0928 08:34:55.487313 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:55.487414 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260928 08:34:55.488406 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:35431
--local_ip_for_outbound_sockets=127.15.134.195
--tserver_master_addrs=127.15.134.254:46653
--webserver_port=45465
--webserver_interface=127.15.134.195
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:34745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:34:55.488478 352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:55.488123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.194:48823)
0928 08:34:55.488287 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:55.488362 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260928 08:34:55.488706 350 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:34:55.828094 353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:34:55.828378 353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:34:55.828476 353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:34:55.835927 353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:34:55.836051 353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:34:55.836189 353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:34:55.846118 353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:34745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:35431
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=45465
--tserver_master_addrs=127.15.134.254:46653
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:34:55.847424 353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:34:55.848733 353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:34:55.862967 359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:34:55.863010 361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:34:55.862927 358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:34:55.864248 353 server_base.cc:1034] running on GCE node
I20260928 08:34:55.864956 353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:34:55.865983 353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:34:55.867223 353 hybrid_clock.cc:648] HybridClock initialized: now 1790584495867163 us; error 46 us; skew 500 ppm
I20260928 08:34:55.867586 353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:34:55 dist-test-slave-qp2n krb5kdc[31369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584495, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:34:55.871320 353 init.cc:377] Logged in from keytab as kudu/127.15.134.195@KRBTEST.COM (short username kudu)
I20260928 08:34:55.873203 353 webserver.cc:460] Webserver started at http://127.15.134.195:45465/ using document root <none> and password file <none>
I20260928 08:34:55.873988 353 fs_manager.cc:359] Metadata directory not provided
I20260928 08:34:55.874140 353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:34:55.880141 353 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 08:34:55.883257 368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:55.884790 353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 08:34:55.884979 353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "a16c6b5ccc094350af081af313e102f3"
format_stamp: "Formatted at 2026-09-28 08:34:10 on dist-test-slave-qp2n"
server_key: "a2437f1368815a68053c4fb84734a1f3"
server_key_iv: "feb16b090e18bb2e3bb3e4a43ee0c910"
server_key_version: "encryptionkey@0"
I20260928 08:34:55.886046 353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:34:55.910246 353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:34:55.915047 353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:34:55.915338 353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:34:55.916380 353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:34:55.917796 353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:34:55.917922 353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:55.918054 353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:34:55.918150 353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:34:55.952839 353 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:35431
I20260928 08:34:55.952927 480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:35431 every 8 connection(s)
I20260928 08:34:55.954329 353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:34:55.959496 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:55.957423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.195:59411)
0928 08:34:55.958229 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:55.958499 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":292,"threads_started":1}
W20260928 08:34:55.960297 481 heartbeater.cc:644] Failed to heartbeat to 127.15.134.254:46653 (0 consecutive failures): Network error: Failed to ping master at 127.15.134.254:46653: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
I20260928 08:34:55.961318 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:55.960912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.195:40367)
0928 08:34:55.961098 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:55.961205 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 08:34:55.962440 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:55.961962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.195:41827)
0928 08:34:55.962221 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:55.962311 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260928 08:34:55.962818 481 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 08:34:55.963932 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 353
I20260928 08:34:56.016748 485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:56.015852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.193:40635)
0928 08:34:56.016334 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:56.016440 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":282,"threads_started":1}
I20260928 08:34:56.490495 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:56.489485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.194:54307)
0928 08:34:56.490066 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:56.490191 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":285,"threads_started":1}
I20260928 08:34:56.964356 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:56.963516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.195:42969)
0928 08:34:56.964009 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:56.964115 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":253,"threads_started":1}
I20260928 08:34:57.018781 488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:57.017785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.193:33139)
0928 08:34:57.018356 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:57.018514 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":360,"threads_started":1}
I20260928 08:34:57.492528 489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:57.491560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.194:37631)
0928 08:34:57.492203 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:57.492315 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":356,"threads_started":1}
I20260928 08:34:57.966157 490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:57.965426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.195:49951)
0928 08:34:57.965889 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:57.965969 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":252,"threads_started":1}
I20260928 08:34:58.020910 491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:58.019964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.193:45823)
0928 08:34:58.020535 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:58.020689 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":256,"threads_started":1}
I20260928 08:34:58.494529 492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:58.493474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.194:42231)
0928 08:34:58.494256 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:58.494334 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":428,"threads_started":1}
I20260928 08:34:58.968142 493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:58.967171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.195:44997)
0928 08:34:58.967721 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:58.967836 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":292,"threads_started":1}
I20260928 08:34:59.022614 494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:59.021878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.193:34003)
0928 08:34:59.022338 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:59.022422 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":238,"threads_started":1}
I20260928 08:34:59.496282 495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:59.495525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.194:53445)
0928 08:34:59.496007 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:59.496103 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":236,"threads_started":1}
I20260928 08:34:59.969841 496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:34:59.969097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.195:57397)
0928 08:34:59.969557 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 08:34:59.969671 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":238,"threads_started":1}
I20260928 08:35:00.024394 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:00.023612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.193:45355)
0928 08:35:00.024142 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:00.024235 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":258,"threads_started":1}
I20260928 08:35:00.498011 498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:00.497247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.194:58501)
0928 08:35:00.497751 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:00.497839 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":270,"threads_started":1}
I20260928 08:35:00.971570 499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:00.970780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.195:44115)
0928 08:35:00.971278 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:00.971364 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":260,"threads_started":1}
I20260928 08:35:01.026294 500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:01.025499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.193:32987)
0928 08:35:01.025961 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:01.026111 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":238,"threads_started":1}
I20260928 08:35:01.499940 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:01.499039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.194:51445)
0928 08:35:01.499602 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:01.499732 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":266,"threads_started":1}
I20260928 08:35:01.973527 502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:01.972701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.195:58897)
0928 08:35:01.973217 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:01.973322 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":260,"threads_started":1}
I20260928 08:35:02.028450 503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:02.027418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.193:59363)
0928 08:35:02.028100 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:02.028217 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":287,"threads_started":1}
I20260928 08:35:02.501956 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:02.501084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.194:35545)
0928 08:35:02.501633 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:02.501732 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":226,"threads_started":1}
I20260928 08:35:02.975670 505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:02.974803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.195:41445)
0928 08:35:02.975340 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:02.975429 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":258,"threads_started":1}
I20260928 08:35:03.030390 506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:03.029448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.193:59465)
0928 08:35:03.030084 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:03.030186 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":283,"threads_started":1}
I20260928 08:35:03.503722 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:03.502942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.194:60305)
0928 08:35:03.503428 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:03.503513 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":236,"threads_started":1}
I20260928 08:35:03.977603 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:03.976851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.195:48893)
0928 08:35:03.977336 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:03.977421 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":260,"threads_started":1}
I20260928 08:35:04.032356 509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:04.031517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.193:53583)
0928 08:35:04.032076 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:04.032189 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":268,"threads_started":1}
I20260928 08:35:04.505705 510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:04.504879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.194:32897)
0928 08:35:04.505408 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:04.505533 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":263,"threads_started":1}
I20260928 08:35:04.979583 511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:04.978559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.195:40993)
0928 08:35:04.979228 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:04.979367 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":253,"threads_started":1}
I20260928 08:35:05.034389 512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:05.033663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.193:54759)
0928 08:35:05.034118 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:05.034203 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":248,"threads_started":1}
I20260928 08:35:05.507480 513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:05.506764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.194:54455)
0928 08:35:05.507227 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:05.507316 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":270,"threads_started":1}
I20260928 08:35:05.981480 514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:05.980706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.195:40229)
0928 08:35:05.981208 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:05.981296 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":277,"threads_started":1}
I20260928 08:35:06.036254 515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:06.035477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.193:46095)
0928 08:35:06.035972 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:06.036073 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":264,"threads_started":1}
I20260928 08:35:06.509297 516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:06.508461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.194:43111)
0928 08:35:06.508970 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:06.509112 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":264,"threads_started":1}
I20260928 08:35:06.983248 517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:06.982467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.195:37651)
0928 08:35:06.982922 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:06.983062 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":238,"threads_started":1}
I20260928 08:35:07.037945 518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:07.037248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.193:52749)
0928 08:35:07.037684 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:07.037772 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":228,"threads_started":1}
I20260928 08:35:07.511157 519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:07.510350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.194:35005)
0928 08:35:07.510865 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:07.510948 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":302,"threads_started":1}
I20260928 08:35:07.985080 520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:07.984209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.195:38713)
0928 08:35:07.984734 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:07.984824 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":250,"threads_started":1}
I20260928 08:35:08.040066 521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:08.039035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.193:38095)
0928 08:35:08.039586 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:08.039739 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":239,"threads_started":1}
I20260928 08:35:08.512940 522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:08.512096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.194:37489)
0928 08:35:08.512586 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:08.512749 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":220,"threads_started":1}
I20260928 08:35:08.986966 523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:08.986136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.195:41729)
0928 08:35:08.986656 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:08.986762 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":248,"threads_started":1}
I20260928 08:35:09.041875 524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:09.041110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.193:43195)
0928 08:35:09.041578 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:09.041678 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":232,"threads_started":1}
I20260928 08:35:09.514554 525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:09.513832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.194:40655)
0928 08:35:09.514253 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:09.514335 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":215,"threads_started":1}
I20260928 08:35:09.989014 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:09.988139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.195:43311)
0928 08:35:09.988671 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:09.988780 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":247,"threads_started":1}
I20260928 08:35:10.044076 527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:10.043233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.193:43325)
0928 08:35:10.043693 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:10.043814 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":255,"threads_started":1}
I20260928 08:35:10.516533 528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:10.515725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.194:46209)
0928 08:35:10.516248 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:10.516345 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":291,"threads_started":1}
I20260928 08:35:10.969298 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 32518
I20260928 08:35:10.991437 530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:10.990342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46653 (local address 127.15.134.195:49381)
0928 08:35:10.990919 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:10.991170 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.254:46653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":296,"threads_started":1}
I20260928 08:35:10.994578 529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:10.970730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43196 (local address 127.15.134.193:43493)
0928 08:35:10.971331 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:10.971347 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:10.971387 (+ 40us) server_negotiation.cc:407] Connection header received
0928 08:35:10.973417 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:10.973430 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:10.973952 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:10.974358 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:10.976152 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:10.977599 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:10.979040 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:10.979800 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:10.980022 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:10.986555 (+ 6533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:10.986565 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:10.986583 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:10.986633 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:10.990248 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:10.991406 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:10.991414 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:10.991422 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:10.991721 (+ 299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:10.992184 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:10.992188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:10.992190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:10.992994 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:10.993291 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:10.993789 (+ 498us) server_negotiation.cc:299] Negotiation successful
0928 08:35:10.994237 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":279,"threads_started":1}
I20260928 08:35:10.996256 32682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:11.086508 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 32518
I20260928 08:35:11.105978 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 32690
I20260928 08:35:11.124296 533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:11.107191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55608 (local address 127.15.134.194:44873)
0928 08:35:11.107808 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:11.107822 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:11.107861 (+ 39us) server_negotiation.cc:407] Connection header received
0928 08:35:11.109568 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:11.109579 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:11.110014 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:11.110326 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:11.111890 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:11.113253 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:11.114604 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:11.115262 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:11.115392 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:11.118114 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:11.118122 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:11.118137 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:11.118185 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:11.121214 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:11.121870 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:11.121876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:11.121882 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:11.122071 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:11.122457 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:11.122461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:11.122463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:11.123016 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:11.123252 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:11.123666 (+ 414us) server_negotiation.cc:299] Negotiation successful
0928 08:35:11.123997 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":313,"threads_started":1}
I20260928 08:35:11.125753 345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:11.212847 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 32690
I20260928 08:35:11.232857 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 353
I20260928 08:35:11.251780 535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:11.234022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51422 (local address 127.15.134.195:35431)
0928 08:35:11.234609 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:11.234623 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:11.234664 (+ 41us) server_negotiation.cc:407] Connection header received
0928 08:35:11.236387 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:11.236397 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:11.236917 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:11.237289 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:11.238744 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:11.240001 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:11.241515 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:11.242182 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:11.242388 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:11.245307 (+ 2919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:11.245315 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:11.245332 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:11.245384 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:11.248313 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:11.249117 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:11.249134 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:11.249143 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:11.249317 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:11.249720 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:11.249724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:11.249726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:11.250263 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:11.250596 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:11.251047 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 08:35:11.251477 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":270,"threads_started":1}
I20260928 08:35:11.253252 476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:11.339017 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 353
2026-09-28T08:35:11Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64759 ms)
[----------] 30 tests from SecurityITest (306020 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:35:11 dist-test-slave-qp2n krb5kdc[539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:35:11 dist-test-slave-qp2n krb5kdc[539](info): set up 2 sockets
Sep 28 08:35:11 dist-test-slave-qp2n krb5kdc[539](info): commencing operation
krb5kdc: starting...
W20260928 08:35:12.474695 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:35:12 dist-test-slave-qp2n krb5kdc[539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:35:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:35:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:35:12.744596 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:46629
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:46629
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:35:13.092518 552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:13.092911 552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:13.093027 552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:13.100589 552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:35:13.100725 552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:13.100772 552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:13.100831 552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:35:13.100873 552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:35:13.111011 552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:46629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:46629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:13.112313 552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:13.113737 552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:13.126164 557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:13.126230 558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:13.126278 560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:13.126818 552 server_base.cc:1034] running on GCE node
I20260928 08:35:13.127650 552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:13.129240 552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:13.130499 552 hybrid_clock.cc:648] HybridClock initialized: now 1790584513130454 us; error 51 us; skew 500 ppm
I20260928 08:35:13.130879 552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:35:13 dist-test-slave-qp2n krb5kdc[539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584513, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:35:13.134485 552 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:35:13.135998 552 webserver.cc:460] Webserver started at http://127.15.134.254:44407/ using document root <none> and password file <none>
I20260928 08:35:13.136783 552 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:13.137014 552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:13.137434 552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:13.141144 552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "ac72be2877a24e6386fb4002f19a2f59"
format_stamp: "Formatted at 2026-09-28 08:35:13 on dist-test-slave-qp2n"
server_key: "e76c5f002aa5f4aebdae3718d58ac1e9"
server_key_iv: "e62e433f79dfd452aef60d2e38b86e85"
server_key_version: "encryptionkey@0"
I20260928 08:35:13.142068 552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "ac72be2877a24e6386fb4002f19a2f59"
format_stamp: "Formatted at 2026-09-28 08:35:13 on dist-test-slave-qp2n"
server_key: "e76c5f002aa5f4aebdae3718d58ac1e9"
server_key_iv: "e62e433f79dfd452aef60d2e38b86e85"
server_key_version: "encryptionkey@0"
I20260928 08:35:13.148005 552 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:35:13.152166 567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:13.153877 552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:35:13.154111 552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "ac72be2877a24e6386fb4002f19a2f59"
format_stamp: "Formatted at 2026-09-28 08:35:13 on dist-test-slave-qp2n"
server_key: "e76c5f002aa5f4aebdae3718d58ac1e9"
server_key_iv: "e62e433f79dfd452aef60d2e38b86e85"
server_key_version: "encryptionkey@0"
I20260928 08:35:13.154340 552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:13.173112 552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:13.177815 552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:13.178139 552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:13.197533 552 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:46629
I20260928 08:35:13.197553 618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:46629 every 8 connection(s)
I20260928 08:35:13.198976 552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:35:13.202167 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 552
I20260928 08:35:13.202397 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:35:13.202878 619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:13.203181 15899 external_mini_cluster.cc:1428] Setting key cd46752a008fde8497841d32ffa0ebc3
I20260928 08:35:13.217621 619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac72be2877a24e6386fb4002f19a2f59: Bootstrap starting.
Sep 28 08:35:13 dist-test-slave-qp2n krb5kdc[539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:35:13.221560 619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac72be2877a24e6386fb4002f19a2f59: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:13.222929 619 log.cc:824] T 00000000000000000000000000000000 P ac72be2877a24e6386fb4002f19a2f59: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:13.226514 619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac72be2877a24e6386fb4002f19a2f59: No bootstrap required, opened a new log
I20260928 08:35:13.228292 622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:13.205124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40534 (local address 127.15.134.254:46629)
0928 08:35:13.205933 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:13.205954 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:13.206000 (+ 46us) server_negotiation.cc:407] Connection header received
0928 08:35:13.207515 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:13.207534 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:13.207906 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:13.208224 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:13.210220 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:13.211561 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:13.213525 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:13.214288 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:13.214425 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:13.219318 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:13.219328 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:13.219355 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:13.219419 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:13.223388 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:13.224338 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:13.224344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:13.224350 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:13.224597 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:13.225048 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:13.225052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:13.225055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:13.225729 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:13.226049 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:13.226453 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 08:35:13.226890 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":381,"threads_started":1}
I20260928 08:35:13.235492 619 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac72be2877a24e6386fb4002f19a2f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac72be2877a24e6386fb4002f19a2f59" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46629 } }
I20260928 08:35:13.235838 619 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac72be2877a24e6386fb4002f19a2f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:13.235908 619 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac72be2877a24e6386fb4002f19a2f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac72be2877a24e6386fb4002f19a2f59, State: Initialized, Role: FOLLOWER
I20260928 08:35:13.236474 619 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac72be2877a24e6386fb4002f19a2f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac72be2877a24e6386fb4002f19a2f59" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46629 } }
I20260928 08:35:13.236744 619 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac72be2877a24e6386fb4002f19a2f59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:35:13.236860 619 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac72be2877a24e6386fb4002f19a2f59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:35:13.237000 619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac72be2877a24e6386fb4002f19a2f59 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:13.239555 619 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac72be2877a24e6386fb4002f19a2f59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac72be2877a24e6386fb4002f19a2f59" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46629 } }
I20260928 08:35:13.239989 619 leader_election.cc:302] T 00000000000000000000000000000000 P ac72be2877a24e6386fb4002f19a2f59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac72be2877a24e6386fb4002f19a2f59; no voters:
I20260928 08:35:13.240499 619 leader_election.cc:288] T 00000000000000000000000000000000 P ac72be2877a24e6386fb4002f19a2f59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:35:13.240720 624 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac72be2877a24e6386fb4002f19a2f59 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:35:13.241410 624 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac72be2877a24e6386fb4002f19a2f59 [term 1 LEADER]: Becoming Leader. State: Replica: ac72be2877a24e6386fb4002f19a2f59, State: Running, Role: LEADER
I20260928 08:35:13.241946 624 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac72be2877a24e6386fb4002f19a2f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac72be2877a24e6386fb4002f19a2f59" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46629 } }
I20260928 08:35:13.242686 619 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac72be2877a24e6386fb4002f19a2f59 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:35:13.245208 626 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac72be2877a24e6386fb4002f19a2f59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac72be2877a24e6386fb4002f19a2f59. Latest consensus state: current_term: 1 leader_uuid: "ac72be2877a24e6386fb4002f19a2f59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac72be2877a24e6386fb4002f19a2f59" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46629 } } }
I20260928 08:35:13.245589 625 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac72be2877a24e6386fb4002f19a2f59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac72be2877a24e6386fb4002f19a2f59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac72be2877a24e6386fb4002f19a2f59" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 46629 } } }
I20260928 08:35:13.245769 626 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac72be2877a24e6386fb4002f19a2f59 [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:13.245864 625 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac72be2877a24e6386fb4002f19a2f59 [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:13.247558 631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:35:13.253808 631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:35:13.263512 631 catalog_manager.cc:1347] Generated new cluster ID: 8ea1620ddd6a4972a9fdcbb6ede7c02c
I20260928 08:35:13.263648 631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:35:13.284868 631 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:35:13.286248 631 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:35:13.295122 631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac72be2877a24e6386fb4002f19a2f59: Generated new TSK 0
I20260928 08:35:13.295997 631 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.134.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
I20260928 08:35:13.368023 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:0
--local_ip_for_outbound_sockets=127.15.134.193
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:46629
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:35:13.726846 645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:13.727202 645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:13.727381 645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:13.735060 645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:13.735229 645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:13.735425 645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:35:13.746253 645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:46629
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:13.747709 645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:13.749271 645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:13.764525 650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:13.764539 653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:13.764568 651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:13.765425 645 server_base.cc:1034] running on GCE node
I20260928 08:35:13.766053 645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:13.767259 645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:13.768512 645 hybrid_clock.cc:648] HybridClock initialized: now 1790584513768441 us; error 64 us; skew 500 ppm
I20260928 08:35:13.768939 645 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:35:13 dist-test-slave-qp2n krb5kdc[539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584513, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:35:13.777268 645 init.cc:377] Logged in from keytab as kudu/127.15.134.193@KRBTEST.COM (short username kudu)
I20260928 08:35:13.779464 645 webserver.cc:460] Webserver started at http://127.15.134.193:35799/ using document root <none> and password file <none>
I20260928 08:35:13.781368 645 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:13.781559 645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:13.782018 645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:13.788475 645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-0/data/instance:
uuid: "121c30a73f294c6a8562c2718cdaa493"
format_stamp: "Formatted at 2026-09-28 08:35:13 on dist-test-slave-qp2n"
server_key: "d18451686920e4b513141bd9c00a6889"
server_key_iv: "70b13f85ed8766234a982368ea9b8dae"
server_key_version: "encryptionkey@0"
I20260928 08:35:13.789979 645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance:
uuid: "121c30a73f294c6a8562c2718cdaa493"
format_stamp: "Formatted at 2026-09-28 08:35:13 on dist-test-slave-qp2n"
server_key: "d18451686920e4b513141bd9c00a6889"
server_key_iv: "70b13f85ed8766234a982368ea9b8dae"
server_key_version: "encryptionkey@0"
I20260928 08:35:13.799762 645 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20260928 08:35:13.805325 660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:13.807114 645 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 08:35:13.807330 645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "121c30a73f294c6a8562c2718cdaa493"
format_stamp: "Formatted at 2026-09-28 08:35:13 on dist-test-slave-qp2n"
server_key: "d18451686920e4b513141bd9c00a6889"
server_key_iv: "70b13f85ed8766234a982368ea9b8dae"
server_key_version: "encryptionkey@0"
I20260928 08:35:13.807590 645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:13.826164 645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:13.832001 645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:13.832255 645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:13.833325 645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:35:13.834661 645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:35:13.834734 645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:13.834816 645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:35:13.834860 645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:13.869098 645 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:38353
I20260928 08:35:13.869185 772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:38353 every 8 connection(s)
I20260928 08:35:13.870513 645 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:35:13.878880 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 645
I20260928 08:35:13.879129 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance
I20260928 08:35:13.879845 15899 external_mini_cluster.cc:1428] Setting key fbae7b42430ace9f393e31f3ea2042a3
Sep 28 08:35:13 dist-test-slave-qp2n krb5kdc[539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584513, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:35:13.892388 775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:13.873181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:42763 (local address 127.15.134.254:46629)
0928 08:35:13.873678 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:13.873683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:13.874548 (+ 865us) server_negotiation.cc:407] Connection header received
0928 08:35:13.876439 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:13.876445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:13.876544 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:13.876710 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:13.878240 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:13.878997 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:13.881037 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:13.881559 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:13.881689 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:13.885898 (+ 4209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:13.885904 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:13.885908 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:13.885943 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:13.888683 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:13.889657 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:13.889661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:13.889663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:13.889786 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:13.890235 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:13.890241 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:13.890246 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:13.890646 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:13.890806 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:13.891588 (+ 782us) server_negotiation.cc:299] Negotiation successful
0928 08:35:13.891861 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":371,"thread_start_us":297,"threads_started":1}
I20260928 08:35:13.893114 776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:13.873638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46629 (local address 127.15.134.193:42763)
0928 08:35:13.874373 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:13.874418 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:13.876168 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:13.876945 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:13.876953 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:13.877486 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:35:13.878062 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:13.878081 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:13.879185 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:13.879189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:13.880801 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:13.880818 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:13.881855 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:13.881862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:13.882147 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:13.882845 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:35:13.882870 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:35:13.885632 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:35:13.888824 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:35:13.888836 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:35:13.888856 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:35:13.889535 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:35:13.889904 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:35:13.889912 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:35:13.889917 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:35:13.890123 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:35:13.890779 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:35:13.890785 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:35:13.891196 (+ 411us) client_negotiation.cc:769] Sending connection context
0928 08:35:13.891435 (+ 239us) client_negotiation.cc:240] Negotiation successful
0928 08:35:13.891873 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":302,"threads_started":1}
I20260928 08:35:13.894922 773 heartbeater.cc:347] Connected to a master server at 127.15.134.254:46629
I20260928 08:35:13.895287 773 heartbeater.cc:464] Registering TS with master...
I20260928 08:35:13.896215 773 heartbeater.cc:511] Master 127.15.134.254:46629 requested a full tablet report, sending...
I20260928 08:35:13.898689 584 ts_manager.cc:194] Registered new tserver with Master: 121c30a73f294c6a8562c2718cdaa493 (127.15.134.193:38353)
I20260928 08:35:13.900202 584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.193@KRBTEST.COM'} at 127.15.134.193:42763
WARNING: no policy specified for kudu/127.15.134.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
I20260928 08:35:13.940339 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:0
--local_ip_for_outbound_sockets=127.15.134.194
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:46629
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:35:14.289803 779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:14.290108 779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:14.290275 779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:14.298552 779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:14.298672 779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:14.298888 779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:35:14.309263 779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:46629
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:14.310602 779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:14.311940 779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:14.325441 784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:14.325477 785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:14.325441 787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:14.326117 779 server_base.cc:1034] running on GCE node
I20260928 08:35:14.326741 779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:14.327684 779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:14.328912 779 hybrid_clock.cc:648] HybridClock initialized: now 1790584514328848 us; error 57 us; skew 500 ppm
I20260928 08:35:14.329265 779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:35:14 dist-test-slave-qp2n krb5kdc[539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584514, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:35:14.332365 779 init.cc:377] Logged in from keytab as kudu/127.15.134.194@KRBTEST.COM (short username kudu)
I20260928 08:35:14.333899 779 webserver.cc:460] Webserver started at http://127.15.134.194:45453/ using document root <none> and password file <none>
I20260928 08:35:14.334679 779 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:14.334822 779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:14.335136 779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:14.338106 779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-1/data/instance:
uuid: "7ac5dbcfceac46d9beccc02a659ca9e8"
format_stamp: "Formatted at 2026-09-28 08:35:14 on dist-test-slave-qp2n"
server_key: "62bbc8a141bfcf5d8f3510a7d1683f96"
server_key_iv: "57ac8bf52ed68a6b306452ba73a08ad8"
server_key_version: "encryptionkey@0"
I20260928 08:35:14.338864 779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance:
uuid: "7ac5dbcfceac46d9beccc02a659ca9e8"
format_stamp: "Formatted at 2026-09-28 08:35:14 on dist-test-slave-qp2n"
server_key: "62bbc8a141bfcf5d8f3510a7d1683f96"
server_key_iv: "57ac8bf52ed68a6b306452ba73a08ad8"
server_key_version: "encryptionkey@0"
I20260928 08:35:14.344677 779 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 08:35:14.348273 795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:14.349803 779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:35:14.350023 779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "7ac5dbcfceac46d9beccc02a659ca9e8"
format_stamp: "Formatted at 2026-09-28 08:35:14 on dist-test-slave-qp2n"
server_key: "62bbc8a141bfcf5d8f3510a7d1683f96"
server_key_iv: "57ac8bf52ed68a6b306452ba73a08ad8"
server_key_version: "encryptionkey@0"
I20260928 08:35:14.350230 779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:14.367362 779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:14.371726 779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:14.372041 779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:14.373214 779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:35:14.374608 779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:35:14.374719 779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:14.374871 779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:35:14.374943 779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:14.410530 779 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:44139
I20260928 08:35:14.410564 907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:44139 every 8 connection(s)
I20260928 08:35:14.411832 779 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:35:14.416254 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 779
I20260928 08:35:14.416460 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance
I20260928 08:35:14.417062 15899 external_mini_cluster.cc:1428] Setting key 4891e28b6b95e577a51f3a8dfb4215bc
Sep 28 08:35:14 dist-test-slave-qp2n krb5kdc[539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584514, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:35:14.437090 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:14.414563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:34823 (local address 127.15.134.254:46629)
0928 08:35:14.415001 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:14.415006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:14.415986 (+ 980us) server_negotiation.cc:407] Connection header received
0928 08:35:14.418795 (+ 2809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:14.418800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:14.418948 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:14.419074 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:14.420702 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:14.421981 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:14.424578 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:14.425445 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:14.425593 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:14.429950 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:14.429962 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:14.429965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:14.430008 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:14.433423 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:14.434260 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:14.434268 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:14.434275 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:14.434441 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:14.434899 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:14.434906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:14.434912 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:14.435307 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:14.435461 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:14.436287 (+ 826us) server_negotiation.cc:299] Negotiation successful
0928 08:35:14.436560 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":234,"threads_started":1}
I20260928 08:35:14.437750 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:14.415079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46629 (local address 127.15.134.194:34823)
0928 08:35:14.415782 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:14.415820 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:14.418459 (+ 2639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:14.419395 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:14.419406 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:14.419947 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:35:14.420506 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:14.420522 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:14.422152 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:14.422161 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:14.424353 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:14.424372 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:14.425918 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:14.425928 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:14.426289 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:14.427139 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:35:14.427175 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:35:14.429699 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:35:14.433594 (+ 3895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:35:14.433602 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:35:14.433615 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:35:14.434096 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:35:14.434577 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:35:14.434580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:35:14.434583 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:35:14.434775 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:35:14.435492 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:35:14.435498 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:35:14.435873 (+ 375us) client_negotiation.cc:769] Sending connection context
0928 08:35:14.436138 (+ 265us) client_negotiation.cc:240] Negotiation successful
0928 08:35:14.436542 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":264,"threads_started":1}
I20260928 08:35:14.439311 908 heartbeater.cc:347] Connected to a master server at 127.15.134.254:46629
I20260928 08:35:14.439643 908 heartbeater.cc:464] Registering TS with master...
I20260928 08:35:14.440375 908 heartbeater.cc:511] Master 127.15.134.254:46629 requested a full tablet report, sending...
I20260928 08:35:14.442025 584 ts_manager.cc:194] Registered new tserver with Master: 7ac5dbcfceac46d9beccc02a659ca9e8 (127.15.134.194:44139)
I20260928 08:35:14.443102 584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.194@KRBTEST.COM'} at 127.15.134.194:34823
WARNING: no policy specified for kudu/127.15.134.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
I20260928 08:35:14.480397 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:0
--local_ip_for_outbound_sockets=127.15.134.195
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:46629
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:41441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:35:14.837576 914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:14.837939 914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:14.838124 914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:14.846091 914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:14.846282 914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:14.846465 914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:35:14.857512 914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:41441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:46629
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:14.859146 914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:14.860693 914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:14.874832 919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:14.875324 920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:14.877308 922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:14.877584 914 server_base.cc:1034] running on GCE node
I20260928 08:35:14.878362 914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:14.879446 914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:14.880702 914 hybrid_clock.cc:648] HybridClock initialized: now 1790584514880619 us; error 62 us; skew 500 ppm
I20260928 08:35:14.881063 914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:35:14 dist-test-slave-qp2n krb5kdc[539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584514, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:35:14.884686 914 init.cc:377] Logged in from keytab as kudu/127.15.134.195@KRBTEST.COM (short username kudu)
I20260928 08:35:14.886466 914 webserver.cc:460] Webserver started at http://127.15.134.195:46037/ using document root <none> and password file <none>
I20260928 08:35:14.887274 914 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:14.887416 914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:14.887763 914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:14.890945 914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-2/data/instance:
uuid: "d4bfb8d6aded4116834a7c7dbf547fbd"
format_stamp: "Formatted at 2026-09-28 08:35:14 on dist-test-slave-qp2n"
server_key: "38a9770f5d60d07b6f4faa6494759170"
server_key_iv: "a067afc017fb7f6a8112f584c9a6bae9"
server_key_version: "encryptionkey@0"
I20260928 08:35:14.891722 914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance:
uuid: "d4bfb8d6aded4116834a7c7dbf547fbd"
format_stamp: "Formatted at 2026-09-28 08:35:14 on dist-test-slave-qp2n"
server_key: "38a9770f5d60d07b6f4faa6494759170"
server_key_iv: "a067afc017fb7f6a8112f584c9a6bae9"
server_key_version: "encryptionkey@0"
I20260928 08:35:14.897301 914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 08:35:14.901016 929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:14.902506 914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:35:14.902698 914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "d4bfb8d6aded4116834a7c7dbf547fbd"
format_stamp: "Formatted at 2026-09-28 08:35:14 on dist-test-slave-qp2n"
server_key: "38a9770f5d60d07b6f4faa6494759170"
server_key_iv: "a067afc017fb7f6a8112f584c9a6bae9"
server_key_version: "encryptionkey@0"
I20260928 08:35:14.902943 914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:14.904235 773 heartbeater.cc:503] Master 127.15.134.254:46629 was elected leader, sending a full tablet report...
I20260928 08:35:14.928697 914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:14.934322 914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:14.935034 914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:14.936226 914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:35:14.937794 914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:35:14.937888 914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:14.937988 914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:35:14.938035 914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:14.985241 914 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:33371
I20260928 08:35:14.985296 1041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:33371 every 8 connection(s)
I20260928 08:35:14.986590 914 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:35:14.990829 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 914
I20260928 08:35:14.991042 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance
I20260928 08:35:14.991520 15899 external_mini_cluster.cc:1428] Setting key 12835d25774afa514565804ebe5fbb5a
Sep 28 08:35:14 dist-test-slave-qp2n krb5kdc[539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584514, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:35:15.008329 1044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:14.989343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:54777 (local address 127.15.134.254:46629)
0928 08:35:14.989838 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:14.989843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:14.990896 (+ 1053us) server_negotiation.cc:407] Connection header received
0928 08:35:14.992946 (+ 2050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:14.992951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:14.993059 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:14.993185 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:14.994785 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:14.995583 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:14.997311 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:14.997734 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:14.997875 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:15.001472 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:15.001477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:15.001480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:15.001515 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:15.004274 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:15.005303 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:15.005308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:15.005312 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:15.005459 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:15.006104 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:15.006109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:15.006112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:15.006488 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:15.006639 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:15.007630 (+ 991us) server_negotiation.cc:299] Negotiation successful
0928 08:35:15.007886 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":246,"threads_started":1}
I20260928 08:35:15.009341 1045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:14.989934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:46629 (local address 127.15.134.195:54777)
0928 08:35:14.990721 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:14.990761 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:14.992662 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:14.993435 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:14.993443 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:14.993995 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:35:14.994603 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:14.994622 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:14.995737 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:14.995748 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:14.997117 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:14.997130 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:14.997969 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:14.997977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:14.998226 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:14.998943 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:35:14.998967 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:35:15.001256 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:35:15.004434 (+ 3178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:35:15.004442 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:35:15.004457 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:35:15.005154 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:35:15.005600 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:35:15.005605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:35:15.005609 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:35:15.005870 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:35:15.006634 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:35:15.006643 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:35:15.007084 (+ 441us) client_negotiation.cc:769] Sending connection context
0928 08:35:15.007526 (+ 442us) client_negotiation.cc:240] Negotiation successful
0928 08:35:15.008069 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":291,"threads_started":1}
I20260928 08:35:15.010740 1042 heartbeater.cc:347] Connected to a master server at 127.15.134.254:46629
I20260928 08:35:15.011111 1042 heartbeater.cc:464] Registering TS with master...
I20260928 08:35:15.011803 1042 heartbeater.cc:511] Master 127.15.134.254:46629 requested a full tablet report, sending...
I20260928 08:35:15.013332 584 ts_manager.cc:194] Registered new tserver with Master: d4bfb8d6aded4116834a7c7dbf547fbd (127.15.134.195:33371)
I20260928 08:35:15.014207 584 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.195@KRBTEST.COM'} at 127.15.134.195:54777
I20260928 08:35:15.019562 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:35:15.037472 1044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:15.024213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40536 (local address 127.15.134.254:46629)
0928 08:35:15.024376 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:15.024380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:15.024897 (+ 517us) server_negotiation.cc:407] Connection header received
0928 08:35:15.025043 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:15.025047 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:15.025122 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:15.025207 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:15.026657 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.027475 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.028965 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.029427 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.029532 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:15.032129 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:15.032134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:15.032138 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:15.032176 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:15.034847 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:15.035546 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:15.035550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:15.035552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:15.035671 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:15.036034 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:15.036037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:15.036039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:15.036358 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:15.036541 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:15.037045 (+ 504us) server_negotiation.cc:299] Negotiation successful
0928 08:35:15.037224 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 08:35:15.042071 584 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:35:15.049525 584 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:35:15.072221 1055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:15.063049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:38353 (local address 127.0.0.1:45672)
0928 08:35:15.063866 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:15.063910 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:15.064141 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:15.066087 (+ 1946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:15.066095 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:15.066132 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:15.066496 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:15.066506 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.067828 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.067835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:15.069811 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:15.069826 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.071203 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.071210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:15.071396 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:15.071468 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 08:35:15.071630 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 08:35:15.071831 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":289,"threads_started":1}
I20260928 08:35:15.072207 1056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:15.063355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45672 (local address 127.15.134.193:38353)
0928 08:35:15.065590 (+ 2235us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:15.065598 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:15.065621 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:35:15.065698 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:15.065704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:15.065886 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:15.066082 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:15.066662 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.067694 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.069998 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.070987 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.071203 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:15.071412 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:15.071679 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 08:35:15.071843 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2064,"thread_start_us":1891,"threads_started":1}
I20260928 08:35:15.080971 708 tablet_service.cc:1467] Processing CreateTablet for tablet eacb689c58914d49a31ca37bf0fa5e26 (DEFAULT_TABLE table=test-table [id=e775498a40e046738f8969cb7b5d8174]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:35:15.082656 1057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:15.063880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:33371 (local address 127.0.0.1:47156)
0928 08:35:15.064823 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:15.064846 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:15.064978 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:15.075460 (+ 10482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:15.075463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:15.075490 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:15.075781 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:15.075789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.077627 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.077633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:15.079811 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:15.079828 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.081791 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.081797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:15.081998 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:15.082035 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 08:35:15.082137 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 08:35:15.082302 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"thread_start_us":926,"threads_started":1}
I20260928 08:35:15.082775 708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eacb689c58914d49a31ca37bf0fa5e26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:15.082787 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:15.064538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47156 (local address 127.15.134.195:33371)
0928 08:35:15.074664 (+ 10126us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:15.074676 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:15.074716 (+ 40us) server_negotiation.cc:407] Connection header received
0928 08:35:15.074842 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:15.074854 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:15.075186 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:15.075433 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:15.076051 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.077474 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.079992 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.081518 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.081766 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:15.081950 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:15.082201 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 08:35:15.082388 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9907,"thread_start_us":11187,"threads_started":1}
I20260928 08:35:15.083446 1055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:15.064178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:44139 (local address 127.0.0.1:57230)
0928 08:35:15.072908 (+ 8730us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:15.072924 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:15.073009 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:15.077296 (+ 4287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:15.077302 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:15.077328 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:15.077546 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:15.077561 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.079069 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.079075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:15.080590 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:15.080658 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.082661 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.082666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:15.082856 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:15.082909 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 08:35:15.082982 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 08:35:15.083136 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8641,"mutex_wait_us":1,"thread_start_us":11922,"threads_started":1}
I20260928 08:35:15.084862 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:15.064673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57230 (local address 127.15.134.194:44139)
0928 08:35:15.076695 (+ 12022us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:15.076706 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:15.076737 (+ 31us) server_negotiation.cc:407] Connection header received
0928 08:35:15.076838 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:15.076846 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:15.077094 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:15.077292 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:15.077717 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.078907 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.080846 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.082462 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.084017 (+ 1555us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:15.084180 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:15.084261 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 08:35:15.084453 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11827,"thread_start_us":12252,"threads_started":1}
I20260928 08:35:15.086787 977 tablet_service.cc:1467] Processing CreateTablet for tablet eacb689c58914d49a31ca37bf0fa5e26 (DEFAULT_TABLE table=test-table [id=e775498a40e046738f8969cb7b5d8174]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:35:15.087783 843 tablet_service.cc:1467] Processing CreateTablet for tablet eacb689c58914d49a31ca37bf0fa5e26 (DEFAULT_TABLE table=test-table [id=e775498a40e046738f8969cb7b5d8174]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:35:15.088487 977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eacb689c58914d49a31ca37bf0fa5e26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:15.089017 843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eacb689c58914d49a31ca37bf0fa5e26. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:15.098107 1061 tablet_bootstrap.cc:492] T eacb689c58914d49a31ca37bf0fa5e26 P 121c30a73f294c6a8562c2718cdaa493: Bootstrap starting.
I20260928 08:35:15.102075 1062 tablet_bootstrap.cc:492] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8: Bootstrap starting.
I20260928 08:35:15.102397 1061 tablet_bootstrap.cc:654] T eacb689c58914d49a31ca37bf0fa5e26 P 121c30a73f294c6a8562c2718cdaa493: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:15.103799 1063 tablet_bootstrap.cc:492] T eacb689c58914d49a31ca37bf0fa5e26 P d4bfb8d6aded4116834a7c7dbf547fbd: Bootstrap starting.
I20260928 08:35:15.104111 1061 log.cc:824] T eacb689c58914d49a31ca37bf0fa5e26 P 121c30a73f294c6a8562c2718cdaa493: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:15.106359 1062 tablet_bootstrap.cc:654] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:15.107326 1063 tablet_bootstrap.cc:654] T eacb689c58914d49a31ca37bf0fa5e26 P d4bfb8d6aded4116834a7c7dbf547fbd: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:15.107911 1062 log.cc:824] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:15.108422 1061 tablet_bootstrap.cc:492] T eacb689c58914d49a31ca37bf0fa5e26 P 121c30a73f294c6a8562c2718cdaa493: No bootstrap required, opened a new log
I20260928 08:35:15.108755 1061 ts_tablet_manager.cc:1397] T eacb689c58914d49a31ca37bf0fa5e26 P 121c30a73f294c6a8562c2718cdaa493: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260928 08:35:15.109368 1063 log.cc:824] T eacb689c58914d49a31ca37bf0fa5e26 P d4bfb8d6aded4116834a7c7dbf547fbd: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:15.111260 1062 tablet_bootstrap.cc:492] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8: No bootstrap required, opened a new log
I20260928 08:35:15.111524 1062 ts_tablet_manager.cc:1397] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 08:35:15.112352 1063 tablet_bootstrap.cc:492] T eacb689c58914d49a31ca37bf0fa5e26 P d4bfb8d6aded4116834a7c7dbf547fbd: No bootstrap required, opened a new log
I20260928 08:35:15.112742 1063 ts_tablet_manager.cc:1397] T eacb689c58914d49a31ca37bf0fa5e26 P d4bfb8d6aded4116834a7c7dbf547fbd: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 08:35:15.119253 1062 raft_consensus.cc:348] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c30a73f294c6a8562c2718cdaa493" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38353 } } peers { permanent_uuid: "7ac5dbcfceac46d9beccc02a659ca9e8" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44139 } } peers { permanent_uuid: "d4bfb8d6aded4116834a7c7dbf547fbd" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 33371 } }
I20260928 08:35:15.119647 1062 raft_consensus.cc:374] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:15.119755 1062 raft_consensus.cc:729] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac5dbcfceac46d9beccc02a659ca9e8, State: Initialized, Role: FOLLOWER
I20260928 08:35:15.119616 1061 raft_consensus.cc:348] T eacb689c58914d49a31ca37bf0fa5e26 P 121c30a73f294c6a8562c2718cdaa493 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c30a73f294c6a8562c2718cdaa493" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38353 } } peers { permanent_uuid: "7ac5dbcfceac46d9beccc02a659ca9e8" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44139 } } peers { permanent_uuid: "d4bfb8d6aded4116834a7c7dbf547fbd" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 33371 } }
I20260928 08:35:15.120025 1061 raft_consensus.cc:374] T eacb689c58914d49a31ca37bf0fa5e26 P 121c30a73f294c6a8562c2718cdaa493 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:15.120121 1061 raft_consensus.cc:729] T eacb689c58914d49a31ca37bf0fa5e26 P 121c30a73f294c6a8562c2718cdaa493 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 121c30a73f294c6a8562c2718cdaa493, State: Initialized, Role: FOLLOWER
I20260928 08:35:15.120546 1062 consensus_queue.cc:260] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c30a73f294c6a8562c2718cdaa493" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38353 } } peers { permanent_uuid: "7ac5dbcfceac46d9beccc02a659ca9e8" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44139 } } peers { permanent_uuid: "d4bfb8d6aded4116834a7c7dbf547fbd" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 33371 } }
I20260928 08:35:15.120702 1061 consensus_queue.cc:260] T eacb689c58914d49a31ca37bf0fa5e26 P 121c30a73f294c6a8562c2718cdaa493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c30a73f294c6a8562c2718cdaa493" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38353 } } peers { permanent_uuid: "7ac5dbcfceac46d9beccc02a659ca9e8" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44139 } } peers { permanent_uuid: "d4bfb8d6aded4116834a7c7dbf547fbd" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 33371 } }
I20260928 08:35:15.121847 908 heartbeater.cc:503] Master 127.15.134.254:46629 was elected leader, sending a full tablet report...
I20260928 08:35:15.122345 1062 ts_tablet_manager.cc:1428] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260928 08:35:15.124516 1063 raft_consensus.cc:348] T eacb689c58914d49a31ca37bf0fa5e26 P d4bfb8d6aded4116834a7c7dbf547fbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c30a73f294c6a8562c2718cdaa493" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38353 } } peers { permanent_uuid: "7ac5dbcfceac46d9beccc02a659ca9e8" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44139 } } peers { permanent_uuid: "d4bfb8d6aded4116834a7c7dbf547fbd" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 33371 } }
I20260928 08:35:15.125089 1063 raft_consensus.cc:374] T eacb689c58914d49a31ca37bf0fa5e26 P d4bfb8d6aded4116834a7c7dbf547fbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:15.125212 1063 raft_consensus.cc:729] T eacb689c58914d49a31ca37bf0fa5e26 P d4bfb8d6aded4116834a7c7dbf547fbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4bfb8d6aded4116834a7c7dbf547fbd, State: Initialized, Role: FOLLOWER
I20260928 08:35:15.126029 1063 consensus_queue.cc:260] T eacb689c58914d49a31ca37bf0fa5e26 P d4bfb8d6aded4116834a7c7dbf547fbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c30a73f294c6a8562c2718cdaa493" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38353 } } peers { permanent_uuid: "7ac5dbcfceac46d9beccc02a659ca9e8" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44139 } } peers { permanent_uuid: "d4bfb8d6aded4116834a7c7dbf547fbd" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 33371 } }
I20260928 08:35:15.126963 1061 ts_tablet_manager.cc:1428] T eacb689c58914d49a31ca37bf0fa5e26 P 121c30a73f294c6a8562c2718cdaa493: Time spent starting tablet: real 0.018s user 0.014s sys 0.004s
I20260928 08:35:15.127837 1042 heartbeater.cc:503] Master 127.15.134.254:46629 was elected leader, sending a full tablet report...
I20260928 08:35:15.128409 1063 ts_tablet_manager.cc:1428] T eacb689c58914d49a31ca37bf0fa5e26 P d4bfb8d6aded4116834a7c7dbf547fbd: Time spent starting tablet: real 0.016s user 0.006s sys 0.006s
W20260928 08:35:15.163993 909 tablet.cc:2369] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:35:15.238364 1043 tablet.cc:2369] T eacb689c58914d49a31ca37bf0fa5e26 P d4bfb8d6aded4116834a7c7dbf547fbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:35:15.354316 1067 raft_consensus.cc:482] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:35:15.354723 1067 raft_consensus.cc:504] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c30a73f294c6a8562c2718cdaa493" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38353 } } peers { permanent_uuid: "7ac5dbcfceac46d9beccc02a659ca9e8" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44139 } } peers { permanent_uuid: "d4bfb8d6aded4116834a7c7dbf547fbd" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 33371 } }
I20260928 08:35:15.356761 1067 leader_election.cc:288] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 121c30a73f294c6a8562c2718cdaa493 (127.15.134.193:38353), d4bfb8d6aded4116834a7c7dbf547fbd (127.15.134.195:33371)
I20260928 08:35:15.364293 1056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:15.357246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:59827 (local address 127.15.134.193:38353)
0928 08:35:15.357564 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:15.357574 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:15.358474 (+ 900us) server_negotiation.cc:407] Connection header received
0928 08:35:15.358675 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:15.358684 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:15.358780 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:15.358851 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:15.359528 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.360313 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.362475 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.363277 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.363508 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:15.363596 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:15.363902 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 08:35:15.364034 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:35:15.364363 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:15.357200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:42899 (local address 127.15.134.195:33371)
0928 08:35:15.357470 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:15.357478 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:15.358530 (+ 1052us) server_negotiation.cc:407] Connection header received
0928 08:35:15.358773 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:15.358777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:15.358988 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:15.359098 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:15.359728 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.360489 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.362610 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.363414 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.363604 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:15.363681 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:15.363982 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 08:35:15.364111 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 08:35:15.364537 1071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:15.357250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:38353 (local address 127.15.134.194:59827)
0928 08:35:15.358321 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:15.358342 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:15.358516 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:15.358894 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:15.358899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:15.358922 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:15.359352 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:15.359363 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.360453 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.360459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:15.362273 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:15.362285 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.363557 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.363564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:15.363731 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:15.363760 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 08:35:15.363875 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 08:35:15.364012 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":973,"mutex_wait_us":74,"thread_start_us":555,"threads_started":1}
I20260928 08:35:15.364596 1070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:15.356995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:33371 (local address 127.15.134.194:42899)
0928 08:35:15.358333 (+ 1338us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:15.358357 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:15.358578 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:15.359122 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:15.359125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:15.359145 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:15.359581 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:15.359591 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.360664 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.360671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:15.362452 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:15.362464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.363637 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.363643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:15.363810 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:15.363833 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 08:35:15.363936 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 08:35:15.364067 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1169,"spinlock_wait_cycles":24960,"thread_start_us":894,"threads_started":1}
I20260928 08:35:15.365796 997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eacb689c58914d49a31ca37bf0fa5e26" candidate_uuid: "7ac5dbcfceac46d9beccc02a659ca9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4bfb8d6aded4116834a7c7dbf547fbd" is_pre_election: true
I20260928 08:35:15.365797 728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eacb689c58914d49a31ca37bf0fa5e26" candidate_uuid: "7ac5dbcfceac46d9beccc02a659ca9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121c30a73f294c6a8562c2718cdaa493" is_pre_election: true
I20260928 08:35:15.366420 997 raft_consensus.cc:2454] T eacb689c58914d49a31ca37bf0fa5e26 P d4bfb8d6aded4116834a7c7dbf547fbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac5dbcfceac46d9beccc02a659ca9e8 in term 0.
I20260928 08:35:15.366420 728 raft_consensus.cc:2454] T eacb689c58914d49a31ca37bf0fa5e26 P 121c30a73f294c6a8562c2718cdaa493 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ac5dbcfceac46d9beccc02a659ca9e8 in term 0.
I20260928 08:35:15.367208 798 leader_election.cc:302] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 121c30a73f294c6a8562c2718cdaa493, 7ac5dbcfceac46d9beccc02a659ca9e8, d4bfb8d6aded4116834a7c7dbf547fbd; no voters:
I20260928 08:35:15.367672 1067 raft_consensus.cc:2781] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:35:15.367844 1067 raft_consensus.cc:482] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:35:15.367985 1067 raft_consensus.cc:3037] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:15.370275 1067 raft_consensus.cc:504] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c30a73f294c6a8562c2718cdaa493" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38353 } } peers { permanent_uuid: "7ac5dbcfceac46d9beccc02a659ca9e8" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44139 } } peers { permanent_uuid: "d4bfb8d6aded4116834a7c7dbf547fbd" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 33371 } }
I20260928 08:35:15.370911 1067 leader_election.cc:288] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8 [CANDIDATE]: Term 1 election: Requested vote from peers 121c30a73f294c6a8562c2718cdaa493 (127.15.134.193:38353), d4bfb8d6aded4116834a7c7dbf547fbd (127.15.134.195:33371)
I20260928 08:35:15.371457 997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eacb689c58914d49a31ca37bf0fa5e26" candidate_uuid: "7ac5dbcfceac46d9beccc02a659ca9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4bfb8d6aded4116834a7c7dbf547fbd"
I20260928 08:35:15.371454 728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eacb689c58914d49a31ca37bf0fa5e26" candidate_uuid: "7ac5dbcfceac46d9beccc02a659ca9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "121c30a73f294c6a8562c2718cdaa493"
I20260928 08:35:15.371675 997 raft_consensus.cc:3037] T eacb689c58914d49a31ca37bf0fa5e26 P d4bfb8d6aded4116834a7c7dbf547fbd [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:15.371675 728 raft_consensus.cc:3037] T eacb689c58914d49a31ca37bf0fa5e26 P 121c30a73f294c6a8562c2718cdaa493 [term 0 FOLLOWER]: Advancing to term 1
W20260928 08:35:15.373306 774 tablet.cc:2369] T eacb689c58914d49a31ca37bf0fa5e26 P 121c30a73f294c6a8562c2718cdaa493: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:35:15.375120 728 raft_consensus.cc:2454] T eacb689c58914d49a31ca37bf0fa5e26 P 121c30a73f294c6a8562c2718cdaa493 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac5dbcfceac46d9beccc02a659ca9e8 in term 1.
I20260928 08:35:15.375120 997 raft_consensus.cc:2454] T eacb689c58914d49a31ca37bf0fa5e26 P d4bfb8d6aded4116834a7c7dbf547fbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ac5dbcfceac46d9beccc02a659ca9e8 in term 1.
I20260928 08:35:15.375590 798 leader_election.cc:302] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 121c30a73f294c6a8562c2718cdaa493, 7ac5dbcfceac46d9beccc02a659ca9e8, d4bfb8d6aded4116834a7c7dbf547fbd; no voters:
I20260928 08:35:15.375924 1067 raft_consensus.cc:2781] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:35:15.376403 1067 raft_consensus.cc:686] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8 [term 1 LEADER]: Becoming Leader. State: Replica: 7ac5dbcfceac46d9beccc02a659ca9e8, State: Running, Role: LEADER
I20260928 08:35:15.376993 1067 consensus_queue.cc:237] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c30a73f294c6a8562c2718cdaa493" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38353 } } peers { permanent_uuid: "7ac5dbcfceac46d9beccc02a659ca9e8" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44139 } } peers { permanent_uuid: "d4bfb8d6aded4116834a7c7dbf547fbd" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 33371 } }
I20260928 08:35:15.381712 582 catalog_manager.cc:5495] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ac5dbcfceac46d9beccc02a659ca9e8 (127.15.134.194). New cstate: current_term: 1 leader_uuid: "7ac5dbcfceac46d9beccc02a659ca9e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "121c30a73f294c6a8562c2718cdaa493" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ac5dbcfceac46d9beccc02a659ca9e8" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 44139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4bfb8d6aded4116834a7c7dbf547fbd" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 33371 } health_report { overall_health: UNKNOWN } } }
I20260928 08:35:15.456553 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:15.450741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57232 (local address 127.15.134.194:44139)
0928 08:35:15.450965 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:15.450973 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:15.450994 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:35:15.451259 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:15.451263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:15.451353 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:15.451509 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:35:15.451900 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.452710 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.454250 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.454677 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.454868 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:15.455400 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:35:15.455918 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:35:15.456113 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:15.456186 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 08:35:15.456269 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 08:35:15.465153 997 raft_consensus.cc:1261] T eacb689c58914d49a31ca37bf0fa5e26 P d4bfb8d6aded4116834a7c7dbf547fbd [term 1 FOLLOWER]: Refusing update from remote peer 7ac5dbcfceac46d9beccc02a659ca9e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:35:15.465162 728 raft_consensus.cc:1261] T eacb689c58914d49a31ca37bf0fa5e26 P 121c30a73f294c6a8562c2718cdaa493 [term 1 FOLLOWER]: Refusing update from remote peer 7ac5dbcfceac46d9beccc02a659ca9e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:35:15.466145 1072 consensus_queue.cc:1035] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "121c30a73f294c6a8562c2718cdaa493" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 38353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:35:15.466501 1067 consensus_queue.cc:1035] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4bfb8d6aded4116834a7c7dbf547fbd" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 33371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:35:15.478228 1075 mvcc.cc:205] Tried to move back new op lower bound from 7334234175334699008 to 7334234174987972608. Current Snapshot: MvccSnapshot[applied={T|T < 7334234175334699008}]
I20260928 08:35:15.492269 582 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:35:15.492668 582 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:35:15.496697 582 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac72be2877a24e6386fb4002f19a2f59: Sending DeleteTablet for 3 replicas of tablet eacb689c58914d49a31ca37bf0fa5e26
I20260928 08:35:15.498088 977 tablet_service.cc:1514] Processing DeleteTablet for tablet eacb689c58914d49a31ca37bf0fa5e26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:35:15 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:47156
I20260928 08:35:15.498135 708 tablet_service.cc:1514] Processing DeleteTablet for tablet eacb689c58914d49a31ca37bf0fa5e26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:35:15 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:45672
I20260928 08:35:15.498155 843 tablet_service.cc:1514] Processing DeleteTablet for tablet eacb689c58914d49a31ca37bf0fa5e26 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:35:15 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:57230
I20260928 08:35:15.504220 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 645
I20260928 08:35:15.527956 1087 tablet_replica.cc:330] T eacb689c58914d49a31ca37bf0fa5e26 P 121c30a73f294c6a8562c2718cdaa493: stopping tablet replica
I20260928 08:35:15.528671 1087 raft_consensus.cc:2229] T eacb689c58914d49a31ca37bf0fa5e26 P 121c30a73f294c6a8562c2718cdaa493 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:35:15.529367 1087 raft_consensus.cc:2258] T eacb689c58914d49a31ca37bf0fa5e26 P 121c30a73f294c6a8562c2718cdaa493 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:35:15.537031 1087 ts_tablet_manager.cc:1905] T eacb689c58914d49a31ca37bf0fa5e26 P 121c30a73f294c6a8562c2718cdaa493: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:35:15.549902 1085 tablet_replica.cc:330] T eacb689c58914d49a31ca37bf0fa5e26 P d4bfb8d6aded4116834a7c7dbf547fbd: stopping tablet replica
I20260928 08:35:15.550436 1085 raft_consensus.cc:2229] T eacb689c58914d49a31ca37bf0fa5e26 P d4bfb8d6aded4116834a7c7dbf547fbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:35:15.551020 1085 raft_consensus.cc:2258] T eacb689c58914d49a31ca37bf0fa5e26 P d4bfb8d6aded4116834a7c7dbf547fbd [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 28 08:35:15 dist-test-slave-qp2n krb5kdc[539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.193@KRBTEST.COM
I20260928 08:35:15.553117 1086 tablet_replica.cc:330] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8: stopping tablet replica
I20260928 08:35:15.553650 1086 raft_consensus.cc:2229] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:35:15.554915 1086 raft_consensus.cc:2258] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:35:15.557910 1086 ts_tablet_manager.cc:1905] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:35:15.560073 1085 ts_tablet_manager.cc:1905] T eacb689c58914d49a31ca37bf0fa5e26 P d4bfb8d6aded4116834a7c7dbf547fbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:35:15.563114 1087 ts_tablet_manager.cc:1918] T eacb689c58914d49a31ca37bf0fa5e26 P 121c30a73f294c6a8562c2718cdaa493: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:35:15.563289 1087 log.cc:1196] T eacb689c58914d49a31ca37bf0fa5e26 P 121c30a73f294c6a8562c2718cdaa493: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-0/wal/wals/eacb689c58914d49a31ca37bf0fa5e26
I20260928 08:35:15.563879 1087 ts_tablet_manager.cc:1939] T eacb689c58914d49a31ca37bf0fa5e26 P 121c30a73f294c6a8562c2718cdaa493: Deleting consensus metadata
I20260928 08:35:15.565398 1056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:15.508021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45674 (local address 127.15.134.193:38353)
0928 08:35:15.508272 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:15.508282 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:15.523986 (+ 15704us) server_negotiation.cc:407] Connection header received
0928 08:35:15.524638 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:15.524649 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:15.524768 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:15.524867 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:15.529963 (+ 5096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.531531 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.540745 (+ 9214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.541408 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.541555 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:15.553648 (+ 12093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:15.553673 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:15.553696 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:15.553750 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:15.558060 (+ 4310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:15.561734 (+ 3674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:15.561745 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:15.561750 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:15.561985 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:15.562883 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:15.562888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:15.562892 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:15.563524 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:15.564039 (+ 515us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:15.564751 (+ 712us) server_negotiation.cc:299] Negotiation successful
0928 08:35:15.564994 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 08:35:15.566407 570 catalog_manager.cc:4841] TS 121c30a73f294c6a8562c2718cdaa493 (127.15.134.193:38353): tablet eacb689c58914d49a31ca37bf0fa5e26 (table test-table [id=e775498a40e046738f8969cb7b5d8174]) successfully deleted
I20260928 08:35:15.566879 1085 ts_tablet_manager.cc:1918] T eacb689c58914d49a31ca37bf0fa5e26 P d4bfb8d6aded4116834a7c7dbf547fbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:35:15.567058 1085 log.cc:1196] T eacb689c58914d49a31ca37bf0fa5e26 P d4bfb8d6aded4116834a7c7dbf547fbd: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/eacb689c58914d49a31ca37bf0fa5e26
I20260928 08:35:15.567451 768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:15.575002 1085 ts_tablet_manager.cc:1939] T eacb689c58914d49a31ca37bf0fa5e26 P d4bfb8d6aded4116834a7c7dbf547fbd: Deleting consensus metadata
I20260928 08:35:15.576583 1086 ts_tablet_manager.cc:1918] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:35:15.576808 1086 log.cc:1196] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/eacb689c58914d49a31ca37bf0fa5e26
I20260928 08:35:15.576974 569 catalog_manager.cc:4841] TS d4bfb8d6aded4116834a7c7dbf547fbd (127.15.134.195:33371): tablet eacb689c58914d49a31ca37bf0fa5e26 (table test-table [id=e775498a40e046738f8969cb7b5d8174]) successfully deleted
I20260928 08:35:15.577369 1086 ts_tablet_manager.cc:1939] T eacb689c58914d49a31ca37bf0fa5e26 P 7ac5dbcfceac46d9beccc02a659ca9e8: Deleting consensus metadata
I20260928 08:35:15.579156 570 catalog_manager.cc:4841] TS 7ac5dbcfceac46d9beccc02a659ca9e8 (127.15.134.194:44139): tablet eacb689c58914d49a31ca37bf0fa5e26 (table test-table [id=e775498a40e046738f8969cb7b5d8174]) successfully deleted
I20260928 08:35:15.750015 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 645
I20260928 08:35:15.776417 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 779
Sep 28 08:35:15 dist-test-slave-qp2n krb5kdc[539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.194@KRBTEST.COM
I20260928 08:35:15.800601 1060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:15.777668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57238 (local address 127.15.134.194:44139)
0928 08:35:15.777883 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:15.777889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:15.777937 (+ 48us) server_negotiation.cc:407] Connection header received
0928 08:35:15.778010 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:15.778015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:15.778144 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:15.778275 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:15.780116 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.781688 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.784117 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.784849 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.784982 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:15.791430 (+ 6448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:15.791451 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:15.791474 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:15.791551 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:15.796592 (+ 5041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:15.797676 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:15.797689 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:15.797695 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:15.797958 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:15.798476 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:15.798483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:15.798489 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:15.799036 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:15.799425 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:15.799768 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 08:35:15.800056 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 08:35:15.801447 903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:15.927467 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 779
I20260928 08:35:15.951189 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 914
Sep 28 08:35:15 dist-test-slave-qp2n krb5kdc[539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.195@KRBTEST.COM
I20260928 08:35:15.966602 1091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:15.951982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47168 (local address 127.15.134.195:33371)
0928 08:35:15.952458 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:15.952465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:15.952483 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:35:15.952542 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:15.952545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:15.952675 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:15.952767 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:15.954139 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.954895 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.956208 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:15.956586 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:15.956950 (+ 364us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:15.961468 (+ 4518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:15.961482 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:15.961499 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:15.961531 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:15.964157 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:15.964851 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:15.964858 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:15.964860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:15.965013 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:15.965324 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:15.965327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:15.965329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:15.965671 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:15.965880 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:15.966155 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 08:35:15.966315 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":246,"threads_started":1}
I20260928 08:35:15.967352 1037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:16.058418 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 914
I20260928 08:35:16.076400 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 552
I20260928 08:35:16.077651 614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:16.190110 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 552
2026-09-28T08:35:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4857 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T08:35:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:35:16Z Disabled control of system clock
I20260928 08:35:16.274879 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:45359
--webserver_interface=127.15.134.254
--webserver_port=0
--builtin_ntp_servers=127.15.134.212:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:45359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:35:16.623332 1097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:16.623656 1097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:16.623739 1097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:16.630966 1097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:35:16.631068 1097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:16.631105 1097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:16.631153 1097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:35:16.631199 1097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:35:16.640630 1097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:45359
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:45359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:16.642005 1097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:16.643997 1097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:16.656766 1105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:16.656744 1102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:16.657284 1103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:16.657853 1097 server_base.cc:1034] running on GCE node
I20260928 08:35:16.658658 1097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:16.660195 1097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:16.661425 1097 hybrid_clock.cc:648] HybridClock initialized: now 1790584516661389 us; error 51 us; skew 500 ppm
I20260928 08:35:16.661782 1097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:35:16.667613 1097 webserver.cc:460] Webserver started at http://127.15.134.254:46503/ using document root <none> and password file <none>
I20260928 08:35:16.668340 1097 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:16.668473 1097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:16.668850 1097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:16.671757 1097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "b23800cecc8c465a88231b826d3beb75"
format_stamp: "Formatted at 2026-09-28 08:35:16 on dist-test-slave-qp2n"
server_key: "15110e9b4dde38a4f0cb348a631a5d8a"
server_key_iv: "39605d51b56e040abe3abb4825cd32c0"
server_key_version: "encryptionkey@0"
I20260928 08:35:16.672562 1097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "b23800cecc8c465a88231b826d3beb75"
format_stamp: "Formatted at 2026-09-28 08:35:16 on dist-test-slave-qp2n"
server_key: "15110e9b4dde38a4f0cb348a631a5d8a"
server_key_iv: "39605d51b56e040abe3abb4825cd32c0"
server_key_version: "encryptionkey@0"
I20260928 08:35:16.677912 1097 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 08:35:16.681625 1111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:16.683139 1097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 08:35:16.683346 1097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "b23800cecc8c465a88231b826d3beb75"
format_stamp: "Formatted at 2026-09-28 08:35:16 on dist-test-slave-qp2n"
server_key: "15110e9b4dde38a4f0cb348a631a5d8a"
server_key_iv: "39605d51b56e040abe3abb4825cd32c0"
server_key_version: "encryptionkey@0"
I20260928 08:35:16.683579 1097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:16.697618 1097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:16.698649 1097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:16.699041 1097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:16.718739 1162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:45359 every 8 connection(s)
I20260928 08:35:16.718763 1097 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:45359
I20260928 08:35:16.720256 1097 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:35:16.724371 1163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:16.729048 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 1097
I20260928 08:35:16.729244 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:35:16.729771 15899 external_mini_cluster.cc:1428] Setting key 3f3b24b167f4128edae11ea0493077a0
I20260928 08:35:16.737560 1163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b23800cecc8c465a88231b826d3beb75: Bootstrap starting.
I20260928 08:35:16.741529 1163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b23800cecc8c465a88231b826d3beb75: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:16.742920 1163 log.cc:824] T 00000000000000000000000000000000 P b23800cecc8c465a88231b826d3beb75: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:16.746658 1163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b23800cecc8c465a88231b826d3beb75: No bootstrap required, opened a new log
I20260928 08:35:16.746635 1166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:16.731796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48020 (local address 127.15.134.254:45359)
0928 08:35:16.732949 (+ 1153us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:16.732970 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:16.733030 (+ 60us) server_negotiation.cc:407] Connection header received
0928 08:35:16.735632 (+ 2602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:16.735666 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:16.736362 (+ 696us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:16.736880 (+ 518us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:16.737467 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:16.739531 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:16.741143 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:16.742285 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:16.742472 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:16.743724 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:16.743732 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:16.743748 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:16.744399 (+ 651us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:16.744515 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:16.744543 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:16.744687 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:16.744919 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 08:35:16.745281 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":325,"threads_started":1}
I20260928 08:35:16.754803 1163 raft_consensus.cc:348] T 00000000000000000000000000000000 P b23800cecc8c465a88231b826d3beb75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23800cecc8c465a88231b826d3beb75" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45359 } }
I20260928 08:35:16.755216 1163 raft_consensus.cc:374] T 00000000000000000000000000000000 P b23800cecc8c465a88231b826d3beb75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:16.755303 1163 raft_consensus.cc:729] T 00000000000000000000000000000000 P b23800cecc8c465a88231b826d3beb75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b23800cecc8c465a88231b826d3beb75, State: Initialized, Role: FOLLOWER
I20260928 08:35:16.755826 1163 consensus_queue.cc:260] T 00000000000000000000000000000000 P b23800cecc8c465a88231b826d3beb75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23800cecc8c465a88231b826d3beb75" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45359 } }
I20260928 08:35:16.756048 1163 raft_consensus.cc:388] T 00000000000000000000000000000000 P b23800cecc8c465a88231b826d3beb75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:35:16.756156 1163 raft_consensus.cc:482] T 00000000000000000000000000000000 P b23800cecc8c465a88231b826d3beb75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:35:16.756294 1163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b23800cecc8c465a88231b826d3beb75 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:16.758620 1163 raft_consensus.cc:504] T 00000000000000000000000000000000 P b23800cecc8c465a88231b826d3beb75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23800cecc8c465a88231b826d3beb75" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45359 } }
I20260928 08:35:16.759238 1163 leader_election.cc:302] T 00000000000000000000000000000000 P b23800cecc8c465a88231b826d3beb75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b23800cecc8c465a88231b826d3beb75; no voters:
I20260928 08:35:16.759976 1163 leader_election.cc:288] T 00000000000000000000000000000000 P b23800cecc8c465a88231b826d3beb75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:35:16.760105 1168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b23800cecc8c465a88231b826d3beb75 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:35:16.760936 1168 raft_consensus.cc:686] T 00000000000000000000000000000000 P b23800cecc8c465a88231b826d3beb75 [term 1 LEADER]: Becoming Leader. State: Replica: b23800cecc8c465a88231b826d3beb75, State: Running, Role: LEADER
I20260928 08:35:16.761530 1168 consensus_queue.cc:237] T 00000000000000000000000000000000 P b23800cecc8c465a88231b826d3beb75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23800cecc8c465a88231b826d3beb75" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45359 } }
I20260928 08:35:16.762564 1163 sys_catalog.cc:564] T 00000000000000000000000000000000 P b23800cecc8c465a88231b826d3beb75 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:35:16.765129 1170 sys_catalog.cc:455] T 00000000000000000000000000000000 P b23800cecc8c465a88231b826d3beb75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b23800cecc8c465a88231b826d3beb75. Latest consensus state: current_term: 1 leader_uuid: "b23800cecc8c465a88231b826d3beb75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23800cecc8c465a88231b826d3beb75" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45359 } } }
I20260928 08:35:16.766292 1170 sys_catalog.cc:458] T 00000000000000000000000000000000 P b23800cecc8c465a88231b826d3beb75 [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:16.764730 1169 sys_catalog.cc:455] T 00000000000000000000000000000000 P b23800cecc8c465a88231b826d3beb75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b23800cecc8c465a88231b826d3beb75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b23800cecc8c465a88231b826d3beb75" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45359 } } }
I20260928 08:35:16.767027 1169 sys_catalog.cc:458] T 00000000000000000000000000000000 P b23800cecc8c465a88231b826d3beb75 [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:16.768524 1176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:35:16.774380 1176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:35:16.784029 1176 catalog_manager.cc:1347] Generated new cluster ID: d7f3adaaec67474885ce9a7334e2f2ef
I20260928 08:35:16.784168 1176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:35:16.797037 1176 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:35:16.798390 1176 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:35:16.807940 1176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b23800cecc8c465a88231b826d3beb75: Generated new TSK 0
I20260928 08:35:16.808884 1176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:35:16.817094 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:0
--local_ip_for_outbound_sockets=127.15.134.193
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:45359
--builtin_ntp_servers=127.15.134.212:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:35:17.165540 1187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:17.165853 1187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:17.166024 1187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:17.173686 1187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:17.173789 1187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:17.173964 1187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:35:17.183260 1187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:45359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:17.184592 1187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:17.186411 1187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:17.200435 1193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:17.200523 1192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:17.200429 1195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:17.201666 1187 server_base.cc:1034] running on GCE node
I20260928 08:35:17.202356 1187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:17.203413 1187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:17.204685 1187 hybrid_clock.cc:648] HybridClock initialized: now 1790584517204589 us; error 46 us; skew 500 ppm
I20260928 08:35:17.205065 1187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:35:17.207569 1187 webserver.cc:460] Webserver started at http://127.15.134.193:40077/ using document root <none> and password file <none>
I20260928 08:35:17.208285 1187 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:17.208376 1187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:17.208798 1187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:17.211791 1187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-0/data/instance:
uuid: "ce67eb2aaef348038f5eace076110384"
format_stamp: "Formatted at 2026-09-28 08:35:17 on dist-test-slave-qp2n"
server_key: "c625efcb27efdb5299f97c7133acd571"
server_key_iv: "0a6826706529443bf0a18270e8693901"
server_key_version: "encryptionkey@0"
I20260928 08:35:17.212586 1187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance:
uuid: "ce67eb2aaef348038f5eace076110384"
format_stamp: "Formatted at 2026-09-28 08:35:17 on dist-test-slave-qp2n"
server_key: "c625efcb27efdb5299f97c7133acd571"
server_key_iv: "0a6826706529443bf0a18270e8693901"
server_key_version: "encryptionkey@0"
I20260928 08:35:17.217723 1187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:35:17.221208 1201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:17.222476 1187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 08:35:17.222652 1187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "ce67eb2aaef348038f5eace076110384"
format_stamp: "Formatted at 2026-09-28 08:35:17 on dist-test-slave-qp2n"
server_key: "c625efcb27efdb5299f97c7133acd571"
server_key_iv: "0a6826706529443bf0a18270e8693901"
server_key_version: "encryptionkey@0"
I20260928 08:35:17.222869 1187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:17.247887 1187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:17.248826 1187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:17.249168 1187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:17.250155 1187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:35:17.251720 1187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:35:17.251857 1187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:17.252017 1187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:35:17.252115 1187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:17.286721 1187 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:46239
I20260928 08:35:17.286831 1313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:46239 every 8 connection(s)
I20260928 08:35:17.288242 1187 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:35:17.292408 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 1187
I20260928 08:35:17.292634 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance
I20260928 08:35:17.293128 15899 external_mini_cluster.cc:1428] Setting key ec0fc5e10dc5f178b3d3565b1986ff5b
I20260928 08:35:17.295828 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:0
--local_ip_for_outbound_sockets=127.15.134.194
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:45359
--builtin_ntp_servers=127.15.134.212:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 08:35:17.301752 1316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:17.290897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:56871 (local address 127.15.134.254:45359)
0928 08:35:17.291365 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:17.291371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:17.292864 (+ 1493us) server_negotiation.cc:407] Connection header received
0928 08:35:17.294985 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:17.294991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:17.295072 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:17.295200 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:17.296374 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:17.297173 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:17.298745 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:17.299164 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:17.299300 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:17.300856 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:17.300862 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:17.300865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:17.300887 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:17.300935 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:17.300947 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:17.301047 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:17.301274 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 08:35:17.301377 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":218,"threads_started":1}
I20260928 08:35:17.302922 1317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:17.291621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:45359 (local address 127.15.134.193:56871)
0928 08:35:17.292695 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:17.292746 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:17.294682 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:17.295439 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:17.295447 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:17.295542 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:35:17.296205 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:17.296223 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:17.297321 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:17.297332 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:17.298597 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:17.298608 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:17.299429 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:17.299436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:17.299680 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:17.300467 (+ 787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:35:17.300470 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:35:17.300512 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:35:17.300604 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:35:17.301046 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:35:17.301060 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:35:17.301071 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 08:35:17.301264 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 08:35:17.301749 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":282,"threads_started":1}
I20260928 08:35:17.304735 1314 heartbeater.cc:347] Connected to a master server at 127.15.134.254:45359
I20260928 08:35:17.305070 1314 heartbeater.cc:464] Registering TS with master...
I20260928 08:35:17.305894 1314 heartbeater.cc:511] Master 127.15.134.254:45359 requested a full tablet report, sending...
I20260928 08:35:17.308235 1128 ts_manager.cc:194] Registered new tserver with Master: ce67eb2aaef348038f5eace076110384 (127.15.134.193:46239)
I20260928 08:35:17.309789 1128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.134.193:56871
W20260928 08:35:17.633339 1318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:17.633703 1318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:17.633867 1318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:17.641234 1318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:17.641381 1318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:17.641603 1318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:35:17.650946 1318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:45359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:17.652357 1318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:17.654274 1318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:17.667754 1323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:17.667754 1324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:17.667989 1326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:17.669255 1318 server_base.cc:1034] running on GCE node
I20260928 08:35:17.669975 1318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:17.670986 1318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:17.672230 1318 hybrid_clock.cc:648] HybridClock initialized: now 1790584517672190 us; error 32 us; skew 500 ppm
I20260928 08:35:17.672576 1318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:35:17.675076 1318 webserver.cc:460] Webserver started at http://127.15.134.194:42171/ using document root <none> and password file <none>
I20260928 08:35:17.675817 1318 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:17.675953 1318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:17.676270 1318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:17.679248 1318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-1/data/instance:
uuid: "61c85d1989e44f6d80a05fb72bd83d48"
format_stamp: "Formatted at 2026-09-28 08:35:17 on dist-test-slave-qp2n"
server_key: "103e5a97efa005ca427ea729ff6a56ec"
server_key_iv: "63e6501e8020474a1d57185569050b35"
server_key_version: "encryptionkey@0"
I20260928 08:35:17.680069 1318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance:
uuid: "61c85d1989e44f6d80a05fb72bd83d48"
format_stamp: "Formatted at 2026-09-28 08:35:17 on dist-test-slave-qp2n"
server_key: "103e5a97efa005ca427ea729ff6a56ec"
server_key_iv: "63e6501e8020474a1d57185569050b35"
server_key_version: "encryptionkey@0"
I20260928 08:35:17.685145 1318 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 08:35:17.688699 1332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:17.689988 1318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:35:17.690161 1318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "61c85d1989e44f6d80a05fb72bd83d48"
format_stamp: "Formatted at 2026-09-28 08:35:17 on dist-test-slave-qp2n"
server_key: "103e5a97efa005ca427ea729ff6a56ec"
server_key_iv: "63e6501e8020474a1d57185569050b35"
server_key_version: "encryptionkey@0"
I20260928 08:35:17.690387 1318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:17.734153 1318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:17.735028 1318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:17.735338 1318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:17.736390 1318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:35:17.737993 1318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:35:17.738126 1318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:17.738278 1318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:35:17.738368 1318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:17.772790 1318 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:46837
I20260928 08:35:17.772845 1444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:46837 every 8 connection(s)
I20260928 08:35:17.774139 1318 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:35:17.782722 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 1318
I20260928 08:35:17.782963 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance
I20260928 08:35:17.783489 15899 external_mini_cluster.cc:1428] Setting key 3a1470bdc58a2fe068548d03d5407cc6
I20260928 08:35:17.786727 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:0
--local_ip_for_outbound_sockets=127.15.134.195
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:45359
--builtin_ntp_servers=127.15.134.212:46237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 08:35:17.787950 1316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:17.776934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:39401 (local address 127.15.134.254:45359)
0928 08:35:17.777127 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:17.777131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:17.778629 (+ 1498us) server_negotiation.cc:407] Connection header received
0928 08:35:17.780546 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:17.780552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:17.780701 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:17.780817 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:17.781954 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:17.782692 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:17.784455 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:17.785083 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:17.785215 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:17.786835 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:17.786843 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:17.786850 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:17.786880 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:17.786926 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:17.787001 (+ 75us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:17.787108 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:17.787332 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 08:35:17.787464 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 08:35:17.788918 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:17.777524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:45359 (local address 127.15.134.194:39401)
0928 08:35:17.778455 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:17.778492 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:17.780236 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:17.781081 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:17.781091 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:17.781194 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:35:17.781774 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:17.781792 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:17.782863 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:17.782869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:17.784271 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:17.784284 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:17.785336 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:17.785350 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:17.785674 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:17.786420 (+ 746us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:35:17.786423 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:35:17.786479 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:35:17.786608 (+ 129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:35:17.787114 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:35:17.787123 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:35:17.787129 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 08:35:17.787278 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 08:35:17.787618 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":274,"threads_started":1}
I20260928 08:35:17.790453 1445 heartbeater.cc:347] Connected to a master server at 127.15.134.254:45359
I20260928 08:35:17.790830 1445 heartbeater.cc:464] Registering TS with master...
I20260928 08:35:17.791850 1445 heartbeater.cc:511] Master 127.15.134.254:45359 requested a full tablet report, sending...
I20260928 08:35:17.793514 1128 ts_manager.cc:194] Registered new tserver with Master: 61c85d1989e44f6d80a05fb72bd83d48 (127.15.134.194:46837)
I20260928 08:35:17.794469 1128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.134.194:39401
W20260928 08:35:18.129567 1448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:18.129842 1448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:18.129930 1448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:18.137281 1448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:18.137384 1448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:18.137491 1448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:35:18.146736 1448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:46237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:45359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:18.148034 1448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:18.149935 1448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:18.163201 1454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:18.163302 1456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:18.163097 1453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:18.164232 1448 server_base.cc:1034] running on GCE node
I20260928 08:35:18.164950 1448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:18.165948 1448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:18.167212 1448 hybrid_clock.cc:648] HybridClock initialized: now 1790584518167158 us; error 61 us; skew 500 ppm
I20260928 08:35:18.167636 1448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:35:18.170243 1448 webserver.cc:460] Webserver started at http://127.15.134.195:45721/ using document root <none> and password file <none>
I20260928 08:35:18.170982 1448 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:18.171123 1448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:18.171447 1448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:18.174431 1448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-2/data/instance:
uuid: "45cf994aac0e45c2a37dfe7f6cf9731c"
format_stamp: "Formatted at 2026-09-28 08:35:18 on dist-test-slave-qp2n"
server_key: "8e7a98143cdeffc31fbcbe1cd5749437"
server_key_iv: "785622aacebfc8a5c40922bffe201279"
server_key_version: "encryptionkey@0"
I20260928 08:35:18.175225 1448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance:
uuid: "45cf994aac0e45c2a37dfe7f6cf9731c"
format_stamp: "Formatted at 2026-09-28 08:35:18 on dist-test-slave-qp2n"
server_key: "8e7a98143cdeffc31fbcbe1cd5749437"
server_key_iv: "785622aacebfc8a5c40922bffe201279"
server_key_version: "encryptionkey@0"
I20260928 08:35:18.180408 1448 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 08:35:18.184029 1462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:18.185587 1448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:35:18.185767 1448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "45cf994aac0e45c2a37dfe7f6cf9731c"
format_stamp: "Formatted at 2026-09-28 08:35:18 on dist-test-slave-qp2n"
server_key: "8e7a98143cdeffc31fbcbe1cd5749437"
server_key_iv: "785622aacebfc8a5c40922bffe201279"
server_key_version: "encryptionkey@0"
I20260928 08:35:18.185981 1448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:18.201772 1448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:18.202579 1448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:18.202869 1448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:18.203790 1448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:35:18.205490 1448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:35:18.205586 1448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:18.205685 1448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:35:18.205749 1448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:18.239382 1448 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:42577
I20260928 08:35:18.239434 1574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:42577 every 8 connection(s)
I20260928 08:35:18.240815 1448 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:35:18.244858 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 1448
I20260928 08:35:18.245059 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance
I20260928 08:35:18.245556 15899 external_mini_cluster.cc:1428] Setting key a450b23e16f4d5e935969436ff5ebe1d
I20260928 08:35:18.254902 1316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:18.243496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:49841 (local address 127.15.134.254:45359)
0928 08:35:18.243725 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:18.243732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:18.245270 (+ 1538us) server_negotiation.cc:407] Connection header received
0928 08:35:18.247652 (+ 2382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:18.247660 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:18.247765 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:18.247852 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:18.249412 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.250288 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.251972 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.252385 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.252493 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:18.254079 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:18.254085 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:18.254088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:18.254109 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:18.254143 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:18.254155 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:18.254228 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:18.254543 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 08:35:18.254649 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 08:35:18.256054 1577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:18.244036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:45359 (local address 127.15.134.195:49841)
0928 08:35:18.245068 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:18.245143 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:18.247260 (+ 2117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:18.248413 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:18.248421 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:18.248516 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:35:18.249256 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:18.249275 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.250444 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.250449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:18.251820 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:18.251833 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.252587 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.252592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:18.252862 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:18.253736 (+ 874us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:35:18.253739 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:35:18.253782 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:35:18.253879 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:35:18.254255 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:35:18.254292 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:35:18.254301 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 08:35:18.254467 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 08:35:18.254788 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":312,"threads_started":1}
I20260928 08:35:18.257591 1575 heartbeater.cc:347] Connected to a master server at 127.15.134.254:45359
I20260928 08:35:18.257891 1575 heartbeater.cc:464] Registering TS with master...
I20260928 08:35:18.258657 1575 heartbeater.cc:511] Master 127.15.134.254:45359 requested a full tablet report, sending...
I20260928 08:35:18.260332 1127 ts_manager.cc:194] Registered new tserver with Master: 45cf994aac0e45c2a37dfe7f6cf9731c (127.15.134.195:42577)
I20260928 08:35:18.261523 1127 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.134.195:49841
I20260928 08:35:18.261583 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:35:18.272284 1316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:18.266151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39628 (local address 127.15.134.254:45359)
0928 08:35:18.266327 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:18.266332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:18.266807 (+ 475us) server_negotiation.cc:407] Connection header received
0928 08:35:18.266983 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:18.266988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:18.267068 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:18.267157 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:18.267676 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.268479 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.270106 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.270560 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.270660 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:18.271692 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:18.271697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:18.271700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:18.271721 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:18.271755 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:18.271766 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:18.271837 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:18.271970 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 08:35:18.272053 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:35:18.276692 1127 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:35:18.283583 1127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:35:18.309330 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:18.296905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:42577 (local address 127.0.0.1:57848)
0928 08:35:18.297630 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:18.297692 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:18.297868 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:18.298819 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:18.298830 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:18.298871 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:18.299402 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:18.299418 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.302288 (+ 2870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.302297 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:18.305235 (+ 2938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:18.305255 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.308030 (+ 2775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.308039 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:18.308345 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:18.308415 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 08:35:18.308555 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 08:35:18.308843 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":306,"threads_started":1}
I20260928 08:35:18.310618 1588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:18.297129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57848 (local address 127.15.134.195:42577)
0928 08:35:18.297900 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:18.297912 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:18.297992 (+ 80us) server_negotiation.cc:407] Connection header received
0928 08:35:18.298093 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:18.298099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:18.298347 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:18.298538 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:18.299841 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.302097 (+ 2256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.305499 (+ 3402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.307350 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.307576 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:18.307862 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:18.309904 (+ 2042us) server_negotiation.cc:299] Negotiation successful
0928 08:35:18.310145 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":421,"threads_started":1}
I20260928 08:35:18.313074 1314 heartbeater.cc:503] Master 127.15.134.254:45359 was elected leader, sending a full tablet report...
I20260928 08:35:18.315763 1510 tablet_service.cc:1467] Processing CreateTablet for tablet a37f22e855c84909b9767bc80cc818b7 (DEFAULT_TABLE table=test-table [id=24bb2e1e25474be08fa8e63a50c64f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:35:18.317348 1510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37f22e855c84909b9767bc80cc818b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:18.318822 1589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:18.300568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:46239 (local address 127.0.0.1:41918)
0928 08:35:18.310678 (+ 10110us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:18.310736 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:18.310887 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:18.312410 (+ 1523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:18.312418 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:18.312451 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:18.312912 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:18.312933 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.314812 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.314817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:18.316453 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:18.316467 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.318008 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.318016 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:18.318210 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:18.318247 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 08:35:18.318368 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 08:35:18.318512 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9943,"thread_start_us":10762,"threads_started":1}
I20260928 08:35:18.318904 1592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:18.300719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41918 (local address 127.15.134.193:46239)
0928 08:35:18.311324 (+ 10605us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:18.311337 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:18.311375 (+ 38us) server_negotiation.cc:407] Connection header received
0928 08:35:18.311498 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:18.311510 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:18.311831 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:18.312098 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:18.313130 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.314662 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.316692 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.317805 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.318008 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:18.318222 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:18.318408 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 08:35:18.318560 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10422,"thread_start_us":10311,"threads_started":1}
I20260928 08:35:18.321061 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:18.297991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:46837 (local address 127.0.0.1:53920)
0928 08:35:18.309598 (+ 11607us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:18.309617 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:18.309720 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:18.314187 (+ 4467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:18.314191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:18.314218 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:18.314440 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:18.314452 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.316705 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.316710 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:18.318529 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:18.318544 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.320273 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.320282 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:18.320491 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:18.320525 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 08:35:18.320588 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 08:35:18.320736 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11489,"mutex_wait_us":4,"thread_start_us":12997,"threads_started":1}
I20260928 08:35:18.322173 1590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:18.298372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53920 (local address 127.15.134.194:46837)
0928 08:35:18.313564 (+ 15192us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:18.313575 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:18.313604 (+ 29us) server_negotiation.cc:407] Connection header received
0928 08:35:18.313705 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:18.313714 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:18.313982 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:18.314613 (+ 631us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:18.314835 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.316505 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.318716 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.320091 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.321401 (+ 1310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:18.321598 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:18.321661 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 08:35:18.321824 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15004,"thread_start_us":16355,"threads_started":1}
I20260928 08:35:18.322351 1249 tablet_service.cc:1467] Processing CreateTablet for tablet a37f22e855c84909b9767bc80cc818b7 (DEFAULT_TABLE table=test-table [id=24bb2e1e25474be08fa8e63a50c64f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:35:18.323642 1249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37f22e855c84909b9767bc80cc818b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:18.325649 1380 tablet_service.cc:1467] Processing CreateTablet for tablet a37f22e855c84909b9767bc80cc818b7 (DEFAULT_TABLE table=test-table [id=24bb2e1e25474be08fa8e63a50c64f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:35:18.326894 1380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a37f22e855c84909b9767bc80cc818b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:18.330175 1593 tablet_bootstrap.cc:492] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c: Bootstrap starting.
I20260928 08:35:18.333801 1593 tablet_bootstrap.cc:654] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:18.334795 1593 log.cc:824] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:18.337661 1593 tablet_bootstrap.cc:492] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c: No bootstrap required, opened a new log
I20260928 08:35:18.337981 1593 ts_tablet_manager.cc:1397] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 08:35:18.339864 1595 tablet_bootstrap.cc:492] T a37f22e855c84909b9767bc80cc818b7 P ce67eb2aaef348038f5eace076110384: Bootstrap starting.
I20260928 08:35:18.343602 1595 tablet_bootstrap.cc:654] T a37f22e855c84909b9767bc80cc818b7 P ce67eb2aaef348038f5eace076110384: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:18.344511 1596 tablet_bootstrap.cc:492] T a37f22e855c84909b9767bc80cc818b7 P 61c85d1989e44f6d80a05fb72bd83d48: Bootstrap starting.
I20260928 08:35:18.344930 1595 log.cc:824] T a37f22e855c84909b9767bc80cc818b7 P ce67eb2aaef348038f5eace076110384: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:18.348112 1596 tablet_bootstrap.cc:654] T a37f22e855c84909b9767bc80cc818b7 P 61c85d1989e44f6d80a05fb72bd83d48: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:18.348739 1595 tablet_bootstrap.cc:492] T a37f22e855c84909b9767bc80cc818b7 P ce67eb2aaef348038f5eace076110384: No bootstrap required, opened a new log
I20260928 08:35:18.349014 1595 ts_tablet_manager.cc:1397] T a37f22e855c84909b9767bc80cc818b7 P ce67eb2aaef348038f5eace076110384: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260928 08:35:18.348718 1593 raft_consensus.cc:348] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cf994aac0e45c2a37dfe7f6cf9731c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 42577 } } peers { permanent_uuid: "61c85d1989e44f6d80a05fb72bd83d48" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46837 } } peers { permanent_uuid: "ce67eb2aaef348038f5eace076110384" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 46239 } }
I20260928 08:35:18.349228 1593 raft_consensus.cc:374] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:18.349344 1593 raft_consensus.cc:729] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45cf994aac0e45c2a37dfe7f6cf9731c, State: Initialized, Role: FOLLOWER
I20260928 08:35:18.349453 1596 log.cc:824] T a37f22e855c84909b9767bc80cc818b7 P 61c85d1989e44f6d80a05fb72bd83d48: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:18.350104 1593 consensus_queue.cc:260] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cf994aac0e45c2a37dfe7f6cf9731c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 42577 } } peers { permanent_uuid: "61c85d1989e44f6d80a05fb72bd83d48" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46837 } } peers { permanent_uuid: "ce67eb2aaef348038f5eace076110384" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 46239 } }
I20260928 08:35:18.351881 1593 ts_tablet_manager.cc:1428] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260928 08:35:18.352386 1596 tablet_bootstrap.cc:492] T a37f22e855c84909b9767bc80cc818b7 P 61c85d1989e44f6d80a05fb72bd83d48: No bootstrap required, opened a new log
I20260928 08:35:18.352445 1575 heartbeater.cc:503] Master 127.15.134.254:45359 was elected leader, sending a full tablet report...
I20260928 08:35:18.352705 1596 ts_tablet_manager.cc:1397] T a37f22e855c84909b9767bc80cc818b7 P 61c85d1989e44f6d80a05fb72bd83d48: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260928 08:35:18.359781 1595 raft_consensus.cc:348] T a37f22e855c84909b9767bc80cc818b7 P ce67eb2aaef348038f5eace076110384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cf994aac0e45c2a37dfe7f6cf9731c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 42577 } } peers { permanent_uuid: "61c85d1989e44f6d80a05fb72bd83d48" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46837 } } peers { permanent_uuid: "ce67eb2aaef348038f5eace076110384" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 46239 } }
I20260928 08:35:18.360205 1595 raft_consensus.cc:374] T a37f22e855c84909b9767bc80cc818b7 P ce67eb2aaef348038f5eace076110384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:18.360325 1595 raft_consensus.cc:729] T a37f22e855c84909b9767bc80cc818b7 P ce67eb2aaef348038f5eace076110384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce67eb2aaef348038f5eace076110384, State: Initialized, Role: FOLLOWER
I20260928 08:35:18.361021 1595 consensus_queue.cc:260] T a37f22e855c84909b9767bc80cc818b7 P ce67eb2aaef348038f5eace076110384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cf994aac0e45c2a37dfe7f6cf9731c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 42577 } } peers { permanent_uuid: "61c85d1989e44f6d80a05fb72bd83d48" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46837 } } peers { permanent_uuid: "ce67eb2aaef348038f5eace076110384" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 46239 } }
I20260928 08:35:18.361320 1596 raft_consensus.cc:348] T a37f22e855c84909b9767bc80cc818b7 P 61c85d1989e44f6d80a05fb72bd83d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cf994aac0e45c2a37dfe7f6cf9731c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 42577 } } peers { permanent_uuid: "61c85d1989e44f6d80a05fb72bd83d48" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46837 } } peers { permanent_uuid: "ce67eb2aaef348038f5eace076110384" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 46239 } }
I20260928 08:35:18.361709 1596 raft_consensus.cc:374] T a37f22e855c84909b9767bc80cc818b7 P 61c85d1989e44f6d80a05fb72bd83d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:18.361828 1596 raft_consensus.cc:729] T a37f22e855c84909b9767bc80cc818b7 P 61c85d1989e44f6d80a05fb72bd83d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c85d1989e44f6d80a05fb72bd83d48, State: Initialized, Role: FOLLOWER
I20260928 08:35:18.362447 1596 consensus_queue.cc:260] T a37f22e855c84909b9767bc80cc818b7 P 61c85d1989e44f6d80a05fb72bd83d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cf994aac0e45c2a37dfe7f6cf9731c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 42577 } } peers { permanent_uuid: "61c85d1989e44f6d80a05fb72bd83d48" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46837 } } peers { permanent_uuid: "ce67eb2aaef348038f5eace076110384" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 46239 } }
I20260928 08:35:18.362929 1595 ts_tablet_manager.cc:1428] T a37f22e855c84909b9767bc80cc818b7 P ce67eb2aaef348038f5eace076110384: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
I20260928 08:35:18.363915 1445 heartbeater.cc:503] Master 127.15.134.254:45359 was elected leader, sending a full tablet report...
I20260928 08:35:18.364503 1596 ts_tablet_manager.cc:1428] T a37f22e855c84909b9767bc80cc818b7 P 61c85d1989e44f6d80a05fb72bd83d48: Time spent starting tablet: real 0.012s user 0.001s sys 0.009s
I20260928 08:35:18.396512 1599 raft_consensus.cc:482] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:35:18.396854 1599 raft_consensus.cc:504] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cf994aac0e45c2a37dfe7f6cf9731c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 42577 } } peers { permanent_uuid: "61c85d1989e44f6d80a05fb72bd83d48" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46837 } } peers { permanent_uuid: "ce67eb2aaef348038f5eace076110384" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 46239 } }
I20260928 08:35:18.398540 1599 leader_election.cc:288] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61c85d1989e44f6d80a05fb72bd83d48 (127.15.134.194:46837), ce67eb2aaef348038f5eace076110384 (127.15.134.193:46239)
I20260928 08:35:18.406056 1592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:18.399009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:57347 (local address 127.15.134.193:46239)
0928 08:35:18.399326 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:18.399332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:18.399354 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:35:18.399629 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:18.399635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:18.399732 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:18.399821 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:18.400380 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.401449 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.403934 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.405182 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.405348 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:18.405430 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:18.405667 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 08:35:18.405761 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260928 08:35:18.406224 1590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:18.399122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:47523 (local address 127.15.134.194:46837)
0928 08:35:18.399485 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:18.399494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:18.399945 (+ 451us) server_negotiation.cc:407] Connection header received
0928 08:35:18.400123 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:18.400126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:18.400194 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:18.400263 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:18.400844 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.401768 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.404160 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.405361 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.405521 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:18.405564 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:18.405841 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 08:35:18.405970 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260928 08:35:18.406129 1577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:18.398996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:46239 (local address 127.15.134.195:57347)
0928 08:35:18.399233 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:18.399259 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:18.399395 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:18.399904 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:18.399911 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:18.399945 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:18.400198 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:18.400206 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.401577 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.401581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:18.403766 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:18.403781 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.405324 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.405328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:18.405494 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:18.405534 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 08:35:18.405628 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 08:35:18.405767 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":89,"spinlock_wait_cycles":26496}
I20260928 08:35:18.406575 1602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:18.399053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:46837 (local address 127.15.134.195:47523)
0928 08:35:18.399852 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:18.399873 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:18.399992 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:18.400314 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:18.400319 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:18.400346 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:18.400680 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:18.400689 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.401904 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.401914 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:18.403995 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:18.404005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.405535 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.405540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:18.405696 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:18.405723 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 08:35:18.405801 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 08:35:18.405916 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":1,"thread_start_us":466,"threads_started":1}
I20260928 08:35:18.407339 1269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37f22e855c84909b9767bc80cc818b7" candidate_uuid: "45cf994aac0e45c2a37dfe7f6cf9731c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce67eb2aaef348038f5eace076110384" is_pre_election: true
I20260928 08:35:18.407723 1400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37f22e855c84909b9767bc80cc818b7" candidate_uuid: "45cf994aac0e45c2a37dfe7f6cf9731c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61c85d1989e44f6d80a05fb72bd83d48" is_pre_election: true
I20260928 08:35:18.407984 1269 raft_consensus.cc:2454] T a37f22e855c84909b9767bc80cc818b7 P ce67eb2aaef348038f5eace076110384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45cf994aac0e45c2a37dfe7f6cf9731c in term 0.
I20260928 08:35:18.408252 1400 raft_consensus.cc:2454] T a37f22e855c84909b9767bc80cc818b7 P 61c85d1989e44f6d80a05fb72bd83d48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45cf994aac0e45c2a37dfe7f6cf9731c in term 0.
I20260928 08:35:18.408794 1465 leader_election.cc:302] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45cf994aac0e45c2a37dfe7f6cf9731c, ce67eb2aaef348038f5eace076110384; no voters:
I20260928 08:35:18.409225 1599 raft_consensus.cc:2781] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:35:18.409384 1599 raft_consensus.cc:482] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:35:18.409518 1599 raft_consensus.cc:3037] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:18.411672 1599 raft_consensus.cc:504] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cf994aac0e45c2a37dfe7f6cf9731c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 42577 } } peers { permanent_uuid: "61c85d1989e44f6d80a05fb72bd83d48" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46837 } } peers { permanent_uuid: "ce67eb2aaef348038f5eace076110384" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 46239 } }
I20260928 08:35:18.412245 1599 leader_election.cc:288] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c [CANDIDATE]: Term 1 election: Requested vote from peers 61c85d1989e44f6d80a05fb72bd83d48 (127.15.134.194:46837), ce67eb2aaef348038f5eace076110384 (127.15.134.193:46239)
I20260928 08:35:18.412796 1400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37f22e855c84909b9767bc80cc818b7" candidate_uuid: "45cf994aac0e45c2a37dfe7f6cf9731c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61c85d1989e44f6d80a05fb72bd83d48"
I20260928 08:35:18.412796 1269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a37f22e855c84909b9767bc80cc818b7" candidate_uuid: "45cf994aac0e45c2a37dfe7f6cf9731c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce67eb2aaef348038f5eace076110384"
I20260928 08:35:18.413005 1400 raft_consensus.cc:3037] T a37f22e855c84909b9767bc80cc818b7 P 61c85d1989e44f6d80a05fb72bd83d48 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:18.413007 1269 raft_consensus.cc:3037] T a37f22e855c84909b9767bc80cc818b7 P ce67eb2aaef348038f5eace076110384 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:18.416234 1269 raft_consensus.cc:2454] T a37f22e855c84909b9767bc80cc818b7 P ce67eb2aaef348038f5eace076110384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45cf994aac0e45c2a37dfe7f6cf9731c in term 1.
I20260928 08:35:18.416234 1400 raft_consensus.cc:2454] T a37f22e855c84909b9767bc80cc818b7 P 61c85d1989e44f6d80a05fb72bd83d48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45cf994aac0e45c2a37dfe7f6cf9731c in term 1.
I20260928 08:35:18.416642 1464 leader_election.cc:302] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45cf994aac0e45c2a37dfe7f6cf9731c, 61c85d1989e44f6d80a05fb72bd83d48; no voters:
I20260928 08:35:18.416965 1599 raft_consensus.cc:2781] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:35:18.417527 1599 raft_consensus.cc:686] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c [term 1 LEADER]: Becoming Leader. State: Replica: 45cf994aac0e45c2a37dfe7f6cf9731c, State: Running, Role: LEADER
I20260928 08:35:18.418079 1599 consensus_queue.cc:237] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cf994aac0e45c2a37dfe7f6cf9731c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 42577 } } peers { permanent_uuid: "61c85d1989e44f6d80a05fb72bd83d48" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46837 } } peers { permanent_uuid: "ce67eb2aaef348038f5eace076110384" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 46239 } }
I20260928 08:35:18.423970 1127 catalog_manager.cc:5495] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c reported cstate change: term changed from 0 to 1, leader changed from <none> to 45cf994aac0e45c2a37dfe7f6cf9731c (127.15.134.195). New cstate: current_term: 1 leader_uuid: "45cf994aac0e45c2a37dfe7f6cf9731c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45cf994aac0e45c2a37dfe7f6cf9731c" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 42577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61c85d1989e44f6d80a05fb72bd83d48" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce67eb2aaef348038f5eace076110384" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 46239 } health_report { overall_health: UNKNOWN } } }
I20260928 08:35:18.444020 1588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:18.438860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57860 (local address 127.15.134.195:42577)
0928 08:35:18.439082 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:18.439087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:18.439106 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:35:18.439196 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:18.439199 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:18.439273 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:18.439396 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:35:18.439720 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.440451 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.441906 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.442321 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.442471 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:18.442917 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:35:18.443461 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:35:18.443578 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:18.443645 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 08:35:18.443709 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 08:35:18.452286 1400 raft_consensus.cc:1261] T a37f22e855c84909b9767bc80cc818b7 P 61c85d1989e44f6d80a05fb72bd83d48 [term 1 FOLLOWER]: Refusing update from remote peer 45cf994aac0e45c2a37dfe7f6cf9731c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:35:18.452286 1269 raft_consensus.cc:1261] T a37f22e855c84909b9767bc80cc818b7 P ce67eb2aaef348038f5eace076110384 [term 1 FOLLOWER]: Refusing update from remote peer 45cf994aac0e45c2a37dfe7f6cf9731c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:35:18.453301 1603 consensus_queue.cc:1035] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce67eb2aaef348038f5eace076110384" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 46239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:35:18.453681 1599 consensus_queue.cc:1035] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c [LEADER]: Connected to new peer: Peer: permanent_uuid: "61c85d1989e44f6d80a05fb72bd83d48" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:35:18.466233 1607 mvcc.cc:205] Tried to move back new op lower bound from 7334234187570532352 to 7334234187444461568. Current Snapshot: MvccSnapshot[applied={T|T < 7334234187570532352}]
I20260928 08:35:18.479803 1128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:35:18.480050 1128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:35:18.483655 1128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b23800cecc8c465a88231b826d3beb75: Sending DeleteTablet for 3 replicas of tablet a37f22e855c84909b9767bc80cc818b7
I20260928 08:35:18.484813 1380 tablet_service.cc:1514] Processing DeleteTablet for tablet a37f22e855c84909b9767bc80cc818b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:35:18 UTC) from {username='slave'} at 127.0.0.1:53920
I20260928 08:35:18.484992 1510 tablet_service.cc:1514] Processing DeleteTablet for tablet a37f22e855c84909b9767bc80cc818b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:35:18 UTC) from {username='slave'} at 127.0.0.1:57848
I20260928 08:35:18.485149 1249 tablet_service.cc:1514] Processing DeleteTablet for tablet a37f22e855c84909b9767bc80cc818b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:35:18 UTC) from {username='slave'} at 127.0.0.1:41918
I20260928 08:35:18.486840 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 1187
I20260928 08:35:18.520483 1615 tablet_replica.cc:330] T a37f22e855c84909b9767bc80cc818b7 P 61c85d1989e44f6d80a05fb72bd83d48: stopping tablet replica
I20260928 08:35:18.521737 1616 tablet_replica.cc:330] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c: stopping tablet replica
I20260928 08:35:18.521914 1615 raft_consensus.cc:2229] T a37f22e855c84909b9767bc80cc818b7 P 61c85d1989e44f6d80a05fb72bd83d48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:35:18.522245 1616 raft_consensus.cc:2229] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:35:18.522557 1615 raft_consensus.cc:2258] T a37f22e855c84909b9767bc80cc818b7 P 61c85d1989e44f6d80a05fb72bd83d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:35:18.523106 1617 tablet_replica.cc:330] T a37f22e855c84909b9767bc80cc818b7 P ce67eb2aaef348038f5eace076110384: stopping tablet replica
I20260928 08:35:18.523403 1616 raft_consensus.cc:2258] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:35:18.523571 1617 raft_consensus.cc:2229] T a37f22e855c84909b9767bc80cc818b7 P ce67eb2aaef348038f5eace076110384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:35:18.524274 1617 raft_consensus.cc:2258] T a37f22e855c84909b9767bc80cc818b7 P ce67eb2aaef348038f5eace076110384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:35:18.525610 1615 ts_tablet_manager.cc:1905] T a37f22e855c84909b9767bc80cc818b7 P 61c85d1989e44f6d80a05fb72bd83d48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:35:18.525691 1592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:18.488684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41932 (local address 127.15.134.193:46239)
0928 08:35:18.495275 (+ 6591us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:18.495285 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:18.513022 (+ 17737us) server_negotiation.cc:407] Connection header received
0928 08:35:18.513693 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:18.513703 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:18.513887 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:18.514001 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:18.514844 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.516431 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.521226 (+ 4795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.521866 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.522026 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:18.523818 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:18.523831 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:18.523852 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:18.524665 (+ 813us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:18.524730 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:18.524769 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:18.524864 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:18.525211 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 08:35:18.525338 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6394}
I20260928 08:35:18.527210 1309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:18.527499 1616 ts_tablet_manager.cc:1905] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:35:18.533433 1615 ts_tablet_manager.cc:1918] T a37f22e855c84909b9767bc80cc818b7 P 61c85d1989e44f6d80a05fb72bd83d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:35:18.533618 1615 log.cc:1196] T a37f22e855c84909b9767bc80cc818b7 P 61c85d1989e44f6d80a05fb72bd83d48: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/a37f22e855c84909b9767bc80cc818b7
I20260928 08:35:18.534147 1615 ts_tablet_manager.cc:1939] T a37f22e855c84909b9767bc80cc818b7 P 61c85d1989e44f6d80a05fb72bd83d48: Deleting consensus metadata
I20260928 08:35:18.535550 1616 ts_tablet_manager.cc:1918] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:35:18.535719 1616 log.cc:1196] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/a37f22e855c84909b9767bc80cc818b7
I20260928 08:35:18.536113 1113 catalog_manager.cc:4841] TS 61c85d1989e44f6d80a05fb72bd83d48 (127.15.134.194:46837): tablet a37f22e855c84909b9767bc80cc818b7 (table test-table [id=24bb2e1e25474be08fa8e63a50c64f07]) successfully deleted
I20260928 08:35:18.536273 1616 ts_tablet_manager.cc:1939] T a37f22e855c84909b9767bc80cc818b7 P 45cf994aac0e45c2a37dfe7f6cf9731c: Deleting consensus metadata
I20260928 08:35:18.538070 1114 catalog_manager.cc:4841] TS 45cf994aac0e45c2a37dfe7f6cf9731c (127.15.134.195:42577): tablet a37f22e855c84909b9767bc80cc818b7 (table test-table [id=24bb2e1e25474be08fa8e63a50c64f07]) successfully deleted
I20260928 08:35:18.529244 1617 ts_tablet_manager.cc:1905] T a37f22e855c84909b9767bc80cc818b7 P ce67eb2aaef348038f5eace076110384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:35:18.626287 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 1187
W20260928 08:35:18.644461 1114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.134.193:46239 (error 108)
I20260928 08:35:18.645208 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 1318
I20260928 08:35:18.647076 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:18.646551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:46239 (local address 127.0.0.1:41934)
0928 08:35:18.646764 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:18.646884 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:46239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260928 08:35:18.647385 1114 catalog_manager.cc:4584] TS ce67eb2aaef348038f5eace076110384 (127.15.134.193:46239): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a37f22e855c84909b9767bc80cc818b7: Network error: Client connection negotiation failed: client connection to 127.15.134.193:46239: connect: Connection refused (error 111)
I20260928 08:35:18.652971 1590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:18.646103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53928 (local address 127.15.134.194:46837)
0928 08:35:18.646292 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:18.646296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:18.646314 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:35:18.646543 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:18.646549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:18.646669 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:18.646783 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:18.647375 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.648241 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.649776 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.650187 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.650321 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:18.651567 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:18.651581 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:18.651595 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:18.652171 (+ 576us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:18.652221 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:18.652240 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:18.652366 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:18.652486 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 08:35:18.652645 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 08:35:18.653666 1440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:18.701428 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:18.700861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:46239 (local address 127.0.0.1:41950)
0928 08:35:18.701098 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:18.701235 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:46239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 08:35:18.702651 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:18.702201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:46239 (local address 127.0.0.1:41960)
0928 08:35:18.702375 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:18.702466 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:46239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 08:35:18.757825 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 1318
I20260928 08:35:18.763484 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:18.762862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:46239 (local address 127.0.0.1:41974)
0928 08:35:18.763124 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:18.763241 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:46239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 08:35:18.764808 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:18.764309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:46239 (local address 127.0.0.1:41982)
0928 08:35:18.764499 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:18.764594 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:46239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 08:35:18.777114 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 1448
I20260928 08:35:18.785588 1588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:18.777978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57870 (local address 127.15.134.195:42577)
0928 08:35:18.778205 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:18.778211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:18.778234 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:35:18.778344 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:18.778350 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:18.778478 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:18.778600 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:18.778972 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.780167 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.781750 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:18.782440 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:18.782602 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:18.783723 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:18.783736 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:18.783758 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:18.784672 (+ 914us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:18.784738 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:18.784766 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:18.784887 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:18.785025 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 08:35:18.785140 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 08:35:18.786433 1570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:18.845561 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:18.845092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:46239 (local address 127.0.0.1:41986)
0928 08:35:18.845315 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:18.845407 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:46239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 08:35:18.846653 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:18.846229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:46239 (local address 127.0.0.1:41990)
0928 08:35:18.846455 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:18.846535 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:46239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260928 08:35:18.909528 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 1448
I20260928 08:35:18.930424 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 1097
I20260928 08:35:18.931623 1158 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:19.042659 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:19.041763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:46239 (local address 127.0.0.1:41992)
0928 08:35:19.042232 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:19.042380 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:46239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352}
I20260928 08:35:19.043134 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 1097
2026-09-28T08:35:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2846 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:35:19 dist-test-slave-qp2n krb5kdc[1626](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:35:19 dist-test-slave-qp2n krb5kdc[1626](info): set up 2 sockets
Sep 28 08:35:19 dist-test-slave-qp2n krb5kdc[1626](info): commencing operation
krb5kdc: starting...
W20260928 08:35:20.162417 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:35:20 dist-test-slave-qp2n krb5kdc[1626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:35:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:35:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:35:20.424515 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:35643
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:44739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:35643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:35:20.767567 1639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:20.767916 1639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:20.768072 1639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:20.775468 1639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:35:20.775581 1639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:20.775681 1639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:20.775792 1639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:35:20.775882 1639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:35:20.786146 1639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:44739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:35643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:35643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:20.787658 1639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:20.789098 1639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:20.801877 1644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:20.801793 1647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:20.801826 1645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:20.802783 1639 server_base.cc:1034] running on GCE node
I20260928 08:35:20.803632 1639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:20.805227 1639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:20.806504 1639 hybrid_clock.cc:648] HybridClock initialized: now 1790584520806444 us; error 63 us; skew 500 ppm
I20260928 08:35:20.806895 1639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:35:20 dist-test-slave-qp2n krb5kdc[1626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584520, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:35:20.810467 1639 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:35:20.812163 1639 webserver.cc:460] Webserver started at http://127.15.134.254:37489/ using document root <none> and password file <none>
I20260928 08:35:20.813001 1639 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:20.813156 1639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:20.813493 1639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:20.816649 1639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "3ec18e7d4f2144a298c976e95f7d2bd6"
format_stamp: "Formatted at 2026-09-28 08:35:20 on dist-test-slave-qp2n"
server_key: "9a1cc2fb40d73dbee139d72518c3e21f"
server_key_iv: "1a35ace3a51c5f7e30f060212ceeccf8"
server_key_version: "encryptionkey@0"
I20260928 08:35:20.817471 1639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "3ec18e7d4f2144a298c976e95f7d2bd6"
format_stamp: "Formatted at 2026-09-28 08:35:20 on dist-test-slave-qp2n"
server_key: "9a1cc2fb40d73dbee139d72518c3e21f"
server_key_iv: "1a35ace3a51c5f7e30f060212ceeccf8"
server_key_version: "encryptionkey@0"
I20260928 08:35:20.822695 1639 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:35:20.826124 1654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:20.827569 1639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:35:20.827816 1639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "3ec18e7d4f2144a298c976e95f7d2bd6"
format_stamp: "Formatted at 2026-09-28 08:35:20 on dist-test-slave-qp2n"
server_key: "9a1cc2fb40d73dbee139d72518c3e21f"
server_key_iv: "1a35ace3a51c5f7e30f060212ceeccf8"
server_key_version: "encryptionkey@0"
I20260928 08:35:20.828044 1639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:20.847584 1639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:20.852401 1639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:20.852756 1639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:20.872364 1639 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:35643
I20260928 08:35:20.872383 1705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:35643 every 8 connection(s)
I20260928 08:35:20.873894 1639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:35:20.877856 1706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:20.879758 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 1639
I20260928 08:35:20.879973 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:35:20.880510 15899 external_mini_cluster.cc:1428] Setting key b036e8d16afd1794cb13fd0f32e9c835
I20260928 08:35:20.892819 1706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ec18e7d4f2144a298c976e95f7d2bd6: Bootstrap starting.
Sep 28 08:35:20 dist-test-slave-qp2n krb5kdc[1626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:35:20.897100 1706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ec18e7d4f2144a298c976e95f7d2bd6: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:20.898221 1706 log.cc:824] T 00000000000000000000000000000000 P 3ec18e7d4f2144a298c976e95f7d2bd6: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:20.901309 1706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ec18e7d4f2144a298c976e95f7d2bd6: No bootstrap required, opened a new log
I20260928 08:35:20.905084 1709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:20.882538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56612 (local address 127.15.134.254:35643)
0928 08:35:20.883428 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:20.883439 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:20.883481 (+ 42us) server_negotiation.cc:407] Connection header received
0928 08:35:20.885219 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:20.885246 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:20.885648 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:20.885978 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:20.887878 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:20.889333 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:20.891257 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:20.891938 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:20.892072 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:20.897348 (+ 5276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:20.897358 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:20.897375 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:20.897424 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:20.900652 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:20.901379 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:20.901390 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:20.901402 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:20.901616 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:20.902006 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:20.902011 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:20.902015 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:20.902585 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:20.902956 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:20.903424 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 08:35:20.903924 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":442,"threads_started":1}
I20260928 08:35:20.910305 1706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ec18e7d4f2144a298c976e95f7d2bd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ec18e7d4f2144a298c976e95f7d2bd6" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35643 } }
I20260928 08:35:20.910789 1706 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ec18e7d4f2144a298c976e95f7d2bd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:20.910902 1706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ec18e7d4f2144a298c976e95f7d2bd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ec18e7d4f2144a298c976e95f7d2bd6, State: Initialized, Role: FOLLOWER
I20260928 08:35:20.911496 1706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ec18e7d4f2144a298c976e95f7d2bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ec18e7d4f2144a298c976e95f7d2bd6" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35643 } }
I20260928 08:35:20.911744 1706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ec18e7d4f2144a298c976e95f7d2bd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:35:20.911901 1706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ec18e7d4f2144a298c976e95f7d2bd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:35:20.912086 1706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ec18e7d4f2144a298c976e95f7d2bd6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:20.914531 1706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ec18e7d4f2144a298c976e95f7d2bd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ec18e7d4f2144a298c976e95f7d2bd6" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35643 } }
I20260928 08:35:20.915059 1706 leader_election.cc:302] T 00000000000000000000000000000000 P 3ec18e7d4f2144a298c976e95f7d2bd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ec18e7d4f2144a298c976e95f7d2bd6; no voters:
I20260928 08:35:20.915712 1706 leader_election.cc:288] T 00000000000000000000000000000000 P 3ec18e7d4f2144a298c976e95f7d2bd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:35:20.915849 1711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ec18e7d4f2144a298c976e95f7d2bd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:35:20.916597 1711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ec18e7d4f2144a298c976e95f7d2bd6 [term 1 LEADER]: Becoming Leader. State: Replica: 3ec18e7d4f2144a298c976e95f7d2bd6, State: Running, Role: LEADER
I20260928 08:35:20.917260 1711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ec18e7d4f2144a298c976e95f7d2bd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ec18e7d4f2144a298c976e95f7d2bd6" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35643 } }
I20260928 08:35:20.918057 1706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ec18e7d4f2144a298c976e95f7d2bd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:35:20.920545 1712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ec18e7d4f2144a298c976e95f7d2bd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ec18e7d4f2144a298c976e95f7d2bd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ec18e7d4f2144a298c976e95f7d2bd6" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35643 } } }
I20260928 08:35:20.920553 1713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ec18e7d4f2144a298c976e95f7d2bd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ec18e7d4f2144a298c976e95f7d2bd6. Latest consensus state: current_term: 1 leader_uuid: "3ec18e7d4f2144a298c976e95f7d2bd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ec18e7d4f2144a298c976e95f7d2bd6" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35643 } } }
I20260928 08:35:20.921222 1712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ec18e7d4f2144a298c976e95f7d2bd6 [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:20.921222 1713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ec18e7d4f2144a298c976e95f7d2bd6 [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:20.923599 1718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:35:20.929473 1718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:35:20.938705 1718 catalog_manager.cc:1347] Generated new cluster ID: ebb6d6e2484f4efba64e8e8b32bb62e7
I20260928 08:35:20.938834 1718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:35:20.964815 1718 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:35:20.966821 1718 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:35:20.978327 1718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ec18e7d4f2144a298c976e95f7d2bd6: Generated new TSK 0
I20260928 08:35:20.979475 1718 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.134.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
I20260928 08:35:21.052881 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:0
--local_ip_for_outbound_sockets=127.15.134.193
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:35643
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:44739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:35:21.403400 1732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:21.403749 1732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:21.403932 1732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:21.411427 1732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:21.411586 1732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:21.411760 1732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:35:21.421923 1732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:44739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:35643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:21.423312 1732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:21.424695 1732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:21.438068 1738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:21.438129 1740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:21.438151 1737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:21.439476 1732 server_base.cc:1034] running on GCE node
I20260928 08:35:21.440177 1732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:21.441351 1732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:21.442612 1732 hybrid_clock.cc:648] HybridClock initialized: now 1790584521442555 us; error 51 us; skew 500 ppm
I20260928 08:35:21.442972 1732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:35:21 dist-test-slave-qp2n krb5kdc[1626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584521, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:35:21.446491 1732 init.cc:377] Logged in from keytab as kudu/127.15.134.193@KRBTEST.COM (short username kudu)
I20260928 08:35:21.448094 1732 webserver.cc:460] Webserver started at http://127.15.134.193:32833/ using document root <none> and password file <none>
I20260928 08:35:21.449028 1732 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:21.449155 1732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:21.449469 1732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:21.452742 1732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-0/data/instance:
uuid: "59e15008e7a14445b194a5bffe7a41a7"
format_stamp: "Formatted at 2026-09-28 08:35:21 on dist-test-slave-qp2n"
server_key: "b01130e79e346765bea0dd67a789acbe"
server_key_iv: "0766c26fad7d46053b94a9c9f45c11ff"
server_key_version: "encryptionkey@0"
I20260928 08:35:21.453637 1732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance:
uuid: "59e15008e7a14445b194a5bffe7a41a7"
format_stamp: "Formatted at 2026-09-28 08:35:21 on dist-test-slave-qp2n"
server_key: "b01130e79e346765bea0dd67a789acbe"
server_key_iv: "0766c26fad7d46053b94a9c9f45c11ff"
server_key_version: "encryptionkey@0"
I20260928 08:35:21.459671 1732 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 08:35:21.463609 1747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:21.465291 1732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:35:21.465459 1732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "59e15008e7a14445b194a5bffe7a41a7"
format_stamp: "Formatted at 2026-09-28 08:35:21 on dist-test-slave-qp2n"
server_key: "b01130e79e346765bea0dd67a789acbe"
server_key_iv: "0766c26fad7d46053b94a9c9f45c11ff"
server_key_version: "encryptionkey@0"
I20260928 08:35:21.465692 1732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:21.489087 1732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:21.493891 1732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:21.494232 1732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:21.495376 1732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:35:21.496894 1732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:35:21.496985 1732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:21.497139 1732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:35:21.497211 1732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:21.531307 1732 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:34871
I20260928 08:35:21.531410 1859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:34871 every 8 connection(s)
I20260928 08:35:21.532748 1732 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:35:21.542562 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 1732
I20260928 08:35:21.542760 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance
I20260928 08:35:21.543344 15899 external_mini_cluster.cc:1428] Setting key 9a3b1acdb41e4d4f948af74d8da38694
Sep 28 08:35:21 dist-test-slave-qp2n krb5kdc[1626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584521, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:35:21.554250 1862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:21.535500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:42409 (local address 127.15.134.254:35643)
0928 08:35:21.535952 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:21.535958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:21.537065 (+ 1107us) server_negotiation.cc:407] Connection header received
0928 08:35:21.538980 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:21.538986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:21.539096 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:21.539218 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:21.540864 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:21.541607 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:21.543886 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:21.544325 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:21.544471 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:21.548523 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:21.548529 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:21.548532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:21.548565 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:21.551137 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:21.551889 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:21.551900 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:21.551903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:21.552019 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:21.552460 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:21.552466 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:21.552474 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:21.552787 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:21.552944 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:21.553670 (+ 726us) server_negotiation.cc:299] Negotiation successful
0928 08:35:21.553890 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":219,"threads_started":1}
I20260928 08:35:21.555265 1863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:21.536083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:35643 (local address 127.15.134.193:42409)
0928 08:35:21.536857 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:21.536904 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:21.538677 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:21.539480 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:21.539490 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:21.540041 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:35:21.540681 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:21.540714 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:21.541771 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:21.541776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:21.543706 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:21.543721 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:21.544550 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:21.544555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:21.544827 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:21.545629 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:35:21.545661 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:35:21.548292 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:35:21.551293 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:35:21.551302 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:35:21.551320 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:35:21.551757 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:35:21.552131 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:35:21.552136 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:35:21.552140 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:35:21.552330 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:35:21.552924 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:35:21.552932 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:35:21.553262 (+ 330us) client_negotiation.cc:769] Sending connection context
0928 08:35:21.553548 (+ 286us) client_negotiation.cc:240] Negotiation successful
0928 08:35:21.554087 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":270,"threads_started":1}
I20260928 08:35:21.557011 1860 heartbeater.cc:347] Connected to a master server at 127.15.134.254:35643
I20260928 08:35:21.557337 1860 heartbeater.cc:464] Registering TS with master...
I20260928 08:35:21.558147 1860 heartbeater.cc:511] Master 127.15.134.254:35643 requested a full tablet report, sending...
I20260928 08:35:21.560513 1671 ts_manager.cc:194] Registered new tserver with Master: 59e15008e7a14445b194a5bffe7a41a7 (127.15.134.193:34871)
I20260928 08:35:21.562732 1671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.193@KRBTEST.COM'} at 127.15.134.193:42409
WARNING: no policy specified for kudu/127.15.134.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
I20260928 08:35:21.601948 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:0
--local_ip_for_outbound_sockets=127.15.134.194
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:35643
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:44739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:35:21.944818 1866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:21.945139 1866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:21.945313 1866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:21.952502 1866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:21.952677 1866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:21.952872 1866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:35:21.963054 1866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:44739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:35643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:21.964457 1866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:21.965833 1866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:21.979245 1871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:21.979385 1872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:21.979984 1874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:21.980007 1866 server_base.cc:1034] running on GCE node
I20260928 08:35:21.980712 1866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:21.981703 1866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:21.982913 1866 hybrid_clock.cc:648] HybridClock initialized: now 1790584521982867 us; error 42 us; skew 500 ppm
I20260928 08:35:21.983242 1866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:35:21 dist-test-slave-qp2n krb5kdc[1626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584521, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:35:21.991196 1866 init.cc:377] Logged in from keytab as kudu/127.15.134.194@KRBTEST.COM (short username kudu)
I20260928 08:35:21.993021 1866 webserver.cc:460] Webserver started at http://127.15.134.194:41191/ using document root <none> and password file <none>
I20260928 08:35:21.993768 1866 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:21.993952 1866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:21.994282 1866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:21.997457 1866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-1/data/instance:
uuid: "56b4bf6cbd7f40699672b12e264d3c3b"
format_stamp: "Formatted at 2026-09-28 08:35:21 on dist-test-slave-qp2n"
server_key: "d6859e1e219a29d85e3844ba7a858c9b"
server_key_iv: "145c8fcc06059a241541524eae9df6c6"
server_key_version: "encryptionkey@0"
I20260928 08:35:21.998268 1866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance:
uuid: "56b4bf6cbd7f40699672b12e264d3c3b"
format_stamp: "Formatted at 2026-09-28 08:35:21 on dist-test-slave-qp2n"
server_key: "d6859e1e219a29d85e3844ba7a858c9b"
server_key_iv: "145c8fcc06059a241541524eae9df6c6"
server_key_version: "encryptionkey@0"
I20260928 08:35:22.003690 1866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:35:22.007793 1881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:22.009379 1866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 08:35:22.009585 1866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "56b4bf6cbd7f40699672b12e264d3c3b"
format_stamp: "Formatted at 2026-09-28 08:35:21 on dist-test-slave-qp2n"
server_key: "d6859e1e219a29d85e3844ba7a858c9b"
server_key_iv: "145c8fcc06059a241541524eae9df6c6"
server_key_version: "encryptionkey@0"
I20260928 08:35:22.009814 1866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:22.040958 1866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:22.045310 1866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:22.045629 1866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:22.046702 1866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:35:22.048079 1866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:35:22.048182 1866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:22.048336 1866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:35:22.048426 1866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:22.083513 1866 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:35317
I20260928 08:35:22.083554 1994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:35317 every 8 connection(s)
I20260928 08:35:22.085078 1866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:35:22.087791 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 1866
I20260928 08:35:22.088004 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance
I20260928 08:35:22.088551 15899 external_mini_cluster.cc:1428] Setting key fcafb4340bb003f274126e9050afa6b1
Sep 28 08:35:22 dist-test-slave-qp2n krb5kdc[1626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584521, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:35:22.124032 1997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:22.087887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:46371 (local address 127.15.134.254:35643)
0928 08:35:22.088477 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:22.088484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:22.089657 (+ 1173us) server_negotiation.cc:407] Connection header received
0928 08:35:22.092220 (+ 2563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:22.092225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:22.092331 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:22.092464 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:22.104105 (+ 11641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.105586 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.110451 (+ 4865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.110963 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.111099 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:22.116524 (+ 5425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:22.116534 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:22.116543 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:22.116593 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:22.120498 (+ 3905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:22.121451 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:22.121456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:22.121458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:22.121603 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:22.122170 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:22.122175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:22.122177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:22.122471 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:22.122636 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:22.123416 (+ 780us) server_negotiation.cc:299] Negotiation successful
0928 08:35:22.123651 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":239,"threads_started":1}
WARNING: no policy specified for kudu/127.15.134.195@KRBTEST.COM; defaulting to no policy
I20260928 08:35:22.125236 1998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:22.088524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:35643 (local address 127.15.134.194:46371)
0928 08:35:22.089480 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:22.089524 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:22.091892 (+ 2368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:22.092776 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:22.092785 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:22.102976 (+ 10191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:35:22.103771 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:22.103800 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.106177 (+ 2377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.106182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:22.109756 (+ 3574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:22.109791 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.111367 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.111380 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:22.111782 (+ 402us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:22.113051 (+ 1269us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:35:22.113090 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:35:22.116225 (+ 3135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:35:22.120716 (+ 4491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:35:22.120724 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:35:22.120740 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:35:22.121212 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:35:22.121730 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:35:22.121735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:35:22.121738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:35:22.122025 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:35:22.122607 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:35:22.122615 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:35:22.122995 (+ 380us) client_negotiation.cc:769] Sending connection context
0928 08:35:22.123284 (+ 289us) client_negotiation.cc:240] Negotiation successful
0928 08:35:22.123858 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":351,"threads_started":1}
I20260928 08:35:22.126870 1995 heartbeater.cc:347] Connected to a master server at 127.15.134.254:35643
I20260928 08:35:22.127223 1995 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.195@KRBTEST.COM" created.
I20260928 08:35:22.128037 1995 heartbeater.cc:511] Master 127.15.134.254:35643 requested a full tablet report, sending...
I20260928 08:35:22.129660 1671 ts_manager.cc:194] Registered new tserver with Master: 56b4bf6cbd7f40699672b12e264d3c3b (127.15.134.194:35317)
I20260928 08:35:22.130501 1671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.194@KRBTEST.COM'} at 127.15.134.194:46371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
I20260928 08:35:22.158481 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:0
--local_ip_for_outbound_sockets=127.15.134.195
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:35643
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:44739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:35:22.502614 2001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:22.502893 2001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:22.503064 2001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:22.510619 2001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:22.510722 2001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:22.510891 2001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:35:22.521070 2001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:44739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:35643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:22.522336 2001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:22.523629 2001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:22.538997 2006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:22.539101 2007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:22.539104 2009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:22.540329 2001 server_base.cc:1034] running on GCE node
I20260928 08:35:22.541021 2001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:22.542038 2001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:22.543260 2001 hybrid_clock.cc:648] HybridClock initialized: now 1790584522543215 us; error 65 us; skew 500 ppm
I20260928 08:35:22.543612 2001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:35:22 dist-test-slave-qp2n krb5kdc[1626](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584522, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:35:22.546756 2001 init.cc:377] Logged in from keytab as kudu/127.15.134.195@KRBTEST.COM (short username kudu)
I20260928 08:35:22.548179 2001 webserver.cc:460] Webserver started at http://127.15.134.195:46551/ using document root <none> and password file <none>
I20260928 08:35:22.548982 2001 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:22.549129 2001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:22.549454 2001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:22.552420 2001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-2/data/instance:
uuid: "bfbeccfe994f4fa59e5dc255d061c651"
format_stamp: "Formatted at 2026-09-28 08:35:22 on dist-test-slave-qp2n"
server_key: "1e3029fc4091b2d0c37a24779bd7722a"
server_key_iv: "369e413905c672a095b2675fc6c11b58"
server_key_version: "encryptionkey@0"
I20260928 08:35:22.553285 2001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance:
uuid: "bfbeccfe994f4fa59e5dc255d061c651"
format_stamp: "Formatted at 2026-09-28 08:35:22 on dist-test-slave-qp2n"
server_key: "1e3029fc4091b2d0c37a24779bd7722a"
server_key_iv: "369e413905c672a095b2675fc6c11b58"
server_key_version: "encryptionkey@0"
I20260928 08:35:22.558880 2001 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:35:22.562876 2016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:22.564604 2001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:35:22.564895 2001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "bfbeccfe994f4fa59e5dc255d061c651"
format_stamp: "Formatted at 2026-09-28 08:35:22 on dist-test-slave-qp2n"
server_key: "1e3029fc4091b2d0c37a24779bd7722a"
server_key_iv: "369e413905c672a095b2675fc6c11b58"
server_key_version: "encryptionkey@0"
I20260928 08:35:22.565127 2001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:22.566119 1860 heartbeater.cc:503] Master 127.15.134.254:35643 was elected leader, sending a full tablet report...
I20260928 08:35:22.580359 2001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:22.584846 2001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:22.585099 2001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:22.586130 2001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:35:22.587472 2001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:35:22.587548 2001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:22.587632 2001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:35:22.587678 2001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:22.623492 2001 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:44851
I20260928 08:35:22.623592 2128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:44851 every 8 connection(s)
I20260928 08:35:22.624855 2001 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:35:22.634490 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 2001
I20260928 08:35:22.634723 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance
I20260928 08:35:22.635192 15899 external_mini_cluster.cc:1428] Setting key 341a03d66abb98fae9500e5db1fd5800
Sep 28 08:35:22 dist-test-slave-qp2n krb5kdc[1626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584522, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:35:22.645689 2131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:22.627559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:36215 (local address 127.15.134.254:35643)
0928 08:35:22.628048 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:22.628053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:22.629112 (+ 1059us) server_negotiation.cc:407] Connection header received
0928 08:35:22.630871 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:22.630875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:22.630961 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:22.631057 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:22.632662 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.633511 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.635603 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.636188 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.636360 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:22.640162 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:22.640167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:22.640171 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:22.640205 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:22.642782 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:22.643504 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:22.643508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:22.643510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:22.643635 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:22.644067 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:22.644070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:22.644072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:22.644335 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:22.644467 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:22.645217 (+ 750us) server_negotiation.cc:299] Negotiation successful
0928 08:35:22.645388 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":196,"threads_started":1}
I20260928 08:35:22.646660 2132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:22.628126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:35643 (local address 127.15.134.195:36215)
0928 08:35:22.628945 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:22.628993 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:22.630623 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:22.631265 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:22.631273 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:22.631891 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:35:22.632447 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:22.632464 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.633690 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.633695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:22.635434 (+ 1739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:22.635448 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.636448 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.636458 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:22.636799 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:22.637489 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:35:22.637518 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:35:22.639949 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:35:22.642950 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:35:22.642958 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:35:22.642972 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:35:22.643395 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:35:22.643730 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:35:22.643733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:35:22.643735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:35:22.643965 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:35:22.644453 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:35:22.644460 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:35:22.644846 (+ 386us) client_negotiation.cc:769] Sending connection context
0928 08:35:22.645132 (+ 286us) client_negotiation.cc:240] Negotiation successful
0928 08:35:22.645515 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":287,"threads_started":1}
I20260928 08:35:22.648054 2129 heartbeater.cc:347] Connected to a master server at 127.15.134.254:35643
I20260928 08:35:22.648366 2129 heartbeater.cc:464] Registering TS with master...
I20260928 08:35:22.649248 2129 heartbeater.cc:511] Master 127.15.134.254:35643 requested a full tablet report, sending...
I20260928 08:35:22.651069 1671 ts_manager.cc:194] Registered new tserver with Master: bfbeccfe994f4fa59e5dc255d061c651 (127.15.134.195:44851)
I20260928 08:35:22.651862 1671 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.195@KRBTEST.COM'} at 127.15.134.195:36215
I20260928 08:35:22.663594 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:35:22.682585 2131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:22.668395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56624 (local address 127.15.134.254:35643)
0928 08:35:22.668667 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:22.668676 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:22.669096 (+ 420us) server_negotiation.cc:407] Connection header received
0928 08:35:22.669315 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:22.669329 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:22.669408 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:22.669531 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:22.671241 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.672070 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.673600 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.674050 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.674205 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:22.677154 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:22.677161 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:22.677164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:22.677199 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:22.679574 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:22.680416 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:22.680420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:22.680422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:22.680547 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:22.681097 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:22.681101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:22.681103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:22.681405 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:22.681681 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:22.682085 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 08:35:22.682280 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:35:22.686321 1671 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:35:22.693044 1671 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:35:22.717222 2145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:22.705773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35678 (local address 127.15.134.194:35317)
0928 08:35:22.706358 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:22.706366 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:22.707209 (+ 843us) server_negotiation.cc:407] Connection header received
0928 08:35:22.707354 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:22.707362 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:22.707595 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:22.707793 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:22.708832 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.710882 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.714055 (+ 3173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.715624 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.715757 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:22.715926 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:22.716695 (+ 769us) server_negotiation.cc:299] Negotiation successful
0928 08:35:22.716872 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":887,"threads_started":1}
I20260928 08:35:22.717278 2143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:22.705149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43578 (local address 127.15.134.193:34871)
0928 08:35:22.705811 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:22.705821 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:22.706344 (+ 523us) server_negotiation.cc:407] Connection header received
0928 08:35:22.706734 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:22.706746 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:22.707023 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:22.708498 (+ 1475us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:22.708796 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.710830 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.714218 (+ 3388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.715693 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.715828 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:22.715947 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:22.716826 (+ 879us) server_negotiation.cc:299] Negotiation successful
0928 08:35:22.716972 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":322,"threads_started":1}
I20260928 08:35:22.717372 2144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:22.704707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:34871 (local address 127.0.0.1:43578)
0928 08:35:22.706189 (+ 1482us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:22.706227 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:22.706504 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:22.707574 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:22.707581 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:22.707613 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:22.708196 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:22.708218 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.711046 (+ 2828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.711055 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:22.714017 (+ 2962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:22.714035 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.716117 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.716124 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:22.716304 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:22.716453 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 08:35:22.716644 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 08:35:22.716919 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1198,"mutex_wait_us":100,"thread_start_us":2082,"threads_started":1}
I20260928 08:35:22.718889 2142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:22.705106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:35317 (local address 127.0.0.1:35678)
0928 08:35:22.707049 (+ 1943us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:22.707070 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:22.707190 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:22.708036 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:22.708044 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:22.708070 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:22.708650 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:22.708661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.711069 (+ 2408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.711074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:22.713851 (+ 2777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:22.713862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.716150 (+ 2288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.716156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:22.716409 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:22.716471 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 08:35:22.716668 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 08:35:22.718493 (+ 1825us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1813,"mutex_wait_us":151,"thread_start_us":879,"threads_started":1}
I20260928 08:35:22.720958 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:22.707545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:44851 (local address 127.0.0.1:51032)
0928 08:35:22.713217 (+ 5672us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:22.713235 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:22.713352 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:22.713973 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:22.713979 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:22.714102 (+ 123us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:22.714453 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:22.714465 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.716352 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.716359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:22.718346 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:22.718358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.720140 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.720146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:22.720303 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:22.720341 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 08:35:22.720472 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 08:35:22.720609 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5552,"thread_start_us":5330,"threads_started":1}
I20260928 08:35:22.721051 2147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:22.707883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51032 (local address 127.15.134.195:44851)
0928 08:35:22.711552 (+ 3669us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:22.711561 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:22.713341 (+ 1780us) server_negotiation.cc:407] Connection header received
0928 08:35:22.713543 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:22.713551 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:22.713769 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:22.713969 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:22.714695 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.716209 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.718511 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.719936 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.720146 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:22.720311 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:22.720519 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 08:35:22.720702 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3479,"thread_start_us":3688,"threads_started":1}
I20260928 08:35:22.721663 1795 tablet_service.cc:1467] Processing CreateTablet for tablet a83e5ec45c3f4d019c34844c86c68bf6 (DEFAULT_TABLE table=test-table [id=c3294042b48643b3a7aaedeaf86cc807]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:35:22.723279 1795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a83e5ec45c3f4d019c34844c86c68bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:22.722932 1929 tablet_service.cc:1467] Processing CreateTablet for tablet a83e5ec45c3f4d019c34844c86c68bf6 (DEFAULT_TABLE table=test-table [id=c3294042b48643b3a7aaedeaf86cc807]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:35:22.724315 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a83e5ec45c3f4d019c34844c86c68bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:22.724004 2064 tablet_service.cc:1467] Processing CreateTablet for tablet a83e5ec45c3f4d019c34844c86c68bf6 (DEFAULT_TABLE table=test-table [id=c3294042b48643b3a7aaedeaf86cc807]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:35:22.725679 2064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a83e5ec45c3f4d019c34844c86c68bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:22.738413 2148 tablet_bootstrap.cc:492] T a83e5ec45c3f4d019c34844c86c68bf6 P bfbeccfe994f4fa59e5dc255d061c651: Bootstrap starting.
I20260928 08:35:22.743378 2148 tablet_bootstrap.cc:654] T a83e5ec45c3f4d019c34844c86c68bf6 P bfbeccfe994f4fa59e5dc255d061c651: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:22.744464 2149 tablet_bootstrap.cc:492] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7: Bootstrap starting.
I20260928 08:35:22.744987 2148 log.cc:824] T a83e5ec45c3f4d019c34844c86c68bf6 P bfbeccfe994f4fa59e5dc255d061c651: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:22.745471 2150 tablet_bootstrap.cc:492] T a83e5ec45c3f4d019c34844c86c68bf6 P 56b4bf6cbd7f40699672b12e264d3c3b: Bootstrap starting.
I20260928 08:35:22.749078 2149 tablet_bootstrap.cc:654] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:22.749383 2150 tablet_bootstrap.cc:654] T a83e5ec45c3f4d019c34844c86c68bf6 P 56b4bf6cbd7f40699672b12e264d3c3b: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:22.749614 2148 tablet_bootstrap.cc:492] T a83e5ec45c3f4d019c34844c86c68bf6 P bfbeccfe994f4fa59e5dc255d061c651: No bootstrap required, opened a new log
I20260928 08:35:22.750102 2148 ts_tablet_manager.cc:1397] T a83e5ec45c3f4d019c34844c86c68bf6 P bfbeccfe994f4fa59e5dc255d061c651: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260928 08:35:22.750458 2149 log.cc:824] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:22.750959 2150 log.cc:824] T a83e5ec45c3f4d019c34844c86c68bf6 P 56b4bf6cbd7f40699672b12e264d3c3b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:22.753638 2149 tablet_bootstrap.cc:492] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7: No bootstrap required, opened a new log
I20260928 08:35:22.753926 2149 ts_tablet_manager.cc:1397] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 08:35:22.755473 2150 tablet_bootstrap.cc:492] T a83e5ec45c3f4d019c34844c86c68bf6 P 56b4bf6cbd7f40699672b12e264d3c3b: No bootstrap required, opened a new log
I20260928 08:35:22.755769 2150 ts_tablet_manager.cc:1397] T a83e5ec45c3f4d019c34844c86c68bf6 P 56b4bf6cbd7f40699672b12e264d3c3b: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260928 08:35:22.762167 2149 raft_consensus.cc:348] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e15008e7a14445b194a5bffe7a41a7" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 34871 } } peers { permanent_uuid: "56b4bf6cbd7f40699672b12e264d3c3b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35317 } } peers { permanent_uuid: "bfbeccfe994f4fa59e5dc255d061c651" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44851 } }
I20260928 08:35:22.762241 2148 raft_consensus.cc:348] T a83e5ec45c3f4d019c34844c86c68bf6 P bfbeccfe994f4fa59e5dc255d061c651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e15008e7a14445b194a5bffe7a41a7" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 34871 } } peers { permanent_uuid: "56b4bf6cbd7f40699672b12e264d3c3b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35317 } } peers { permanent_uuid: "bfbeccfe994f4fa59e5dc255d061c651" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44851 } }
I20260928 08:35:22.762619 2149 raft_consensus.cc:374] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:22.762710 2149 raft_consensus.cc:729] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59e15008e7a14445b194a5bffe7a41a7, State: Initialized, Role: FOLLOWER
I20260928 08:35:22.762790 2148 raft_consensus.cc:374] T a83e5ec45c3f4d019c34844c86c68bf6 P bfbeccfe994f4fa59e5dc255d061c651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:22.762923 2148 raft_consensus.cc:729] T a83e5ec45c3f4d019c34844c86c68bf6 P bfbeccfe994f4fa59e5dc255d061c651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfbeccfe994f4fa59e5dc255d061c651, State: Initialized, Role: FOLLOWER
I20260928 08:35:22.763562 2149 consensus_queue.cc:260] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e15008e7a14445b194a5bffe7a41a7" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 34871 } } peers { permanent_uuid: "56b4bf6cbd7f40699672b12e264d3c3b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35317 } } peers { permanent_uuid: "bfbeccfe994f4fa59e5dc255d061c651" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44851 } }
I20260928 08:35:22.763595 2148 consensus_queue.cc:260] T a83e5ec45c3f4d019c34844c86c68bf6 P bfbeccfe994f4fa59e5dc255d061c651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e15008e7a14445b194a5bffe7a41a7" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 34871 } } peers { permanent_uuid: "56b4bf6cbd7f40699672b12e264d3c3b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35317 } } peers { permanent_uuid: "bfbeccfe994f4fa59e5dc255d061c651" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44851 } }
I20260928 08:35:22.765614 2149 ts_tablet_manager.cc:1428] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260928 08:35:22.766731 2129 heartbeater.cc:503] Master 127.15.134.254:35643 was elected leader, sending a full tablet report...
I20260928 08:35:22.768939 2150 raft_consensus.cc:348] T a83e5ec45c3f4d019c34844c86c68bf6 P 56b4bf6cbd7f40699672b12e264d3c3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e15008e7a14445b194a5bffe7a41a7" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 34871 } } peers { permanent_uuid: "56b4bf6cbd7f40699672b12e264d3c3b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35317 } } peers { permanent_uuid: "bfbeccfe994f4fa59e5dc255d061c651" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44851 } }
I20260928 08:35:22.769658 2150 raft_consensus.cc:374] T a83e5ec45c3f4d019c34844c86c68bf6 P 56b4bf6cbd7f40699672b12e264d3c3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:22.769847 2150 raft_consensus.cc:729] T a83e5ec45c3f4d019c34844c86c68bf6 P 56b4bf6cbd7f40699672b12e264d3c3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56b4bf6cbd7f40699672b12e264d3c3b, State: Initialized, Role: FOLLOWER
I20260928 08:35:22.769927 2148 ts_tablet_manager.cc:1428] T a83e5ec45c3f4d019c34844c86c68bf6 P bfbeccfe994f4fa59e5dc255d061c651: Time spent starting tablet: real 0.020s user 0.008s sys 0.009s
I20260928 08:35:22.770618 2150 consensus_queue.cc:260] T a83e5ec45c3f4d019c34844c86c68bf6 P 56b4bf6cbd7f40699672b12e264d3c3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e15008e7a14445b194a5bffe7a41a7" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 34871 } } peers { permanent_uuid: "56b4bf6cbd7f40699672b12e264d3c3b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35317 } } peers { permanent_uuid: "bfbeccfe994f4fa59e5dc255d061c651" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44851 } }
I20260928 08:35:22.772054 1995 heartbeater.cc:503] Master 127.15.134.254:35643 was elected leader, sending a full tablet report...
I20260928 08:35:22.772841 2150 ts_tablet_manager.cc:1428] T a83e5ec45c3f4d019c34844c86c68bf6 P 56b4bf6cbd7f40699672b12e264d3c3b: Time spent starting tablet: real 0.017s user 0.014s sys 0.005s
I20260928 08:35:22.777194 2154 raft_consensus.cc:482] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:35:22.777456 2154 raft_consensus.cc:504] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e15008e7a14445b194a5bffe7a41a7" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 34871 } } peers { permanent_uuid: "56b4bf6cbd7f40699672b12e264d3c3b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35317 } } peers { permanent_uuid: "bfbeccfe994f4fa59e5dc255d061c651" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44851 } }
I20260928 08:35:22.779078 2154 leader_election.cc:288] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56b4bf6cbd7f40699672b12e264d3c3b (127.15.134.194:35317), bfbeccfe994f4fa59e5dc255d061c651 (127.15.134.195:44851)
W20260928 08:35:22.785223 1861 tablet.cc:2369] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:35:22.786480 2147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:22.779671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:42645 (local address 127.15.134.195:44851)
0928 08:35:22.779955 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:22.779964 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:22.780798 (+ 834us) server_negotiation.cc:407] Connection header received
0928 08:35:22.780977 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:22.780983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:22.781063 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:22.781168 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:22.781774 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.782608 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.784422 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.785414 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.785600 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:22.785690 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:22.786045 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 08:35:22.786179 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 08:35:22.786674 2157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:22.779576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:44851 (local address 127.15.134.193:42645)
0928 08:35:22.780710 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:22.780725 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:22.780831 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:22.781182 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:22.781187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:22.781211 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:22.781597 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:22.781609 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.782769 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.782773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:22.784252 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:22.784263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.785631 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.785638 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:22.785873 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:22.785900 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 08:35:22.786018 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 08:35:22.786165 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":979,"spinlock_wait_cycles":896,"thread_start_us":608,"threads_started":1}
I20260928 08:35:22.786870 2145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:22.779487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:47145 (local address 127.15.134.194:35317)
0928 08:35:22.779913 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:22.779928 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:22.780713 (+ 785us) server_negotiation.cc:407] Connection header received
0928 08:35:22.781002 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:22.781005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:22.781066 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:22.781162 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:22.781897 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.782602 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.784577 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.785752 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.785904 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:22.785986 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:22.786447 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 08:35:22.786586 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260928 08:35:22.787025 2158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:22.779440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:35317 (local address 127.15.134.193:47145)
0928 08:35:22.780578 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:22.780595 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:22.780815 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:22.781182 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:22.781187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:22.781277 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:22.781734 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:22.781749 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.782747 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.782756 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:22.784424 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:22.784439 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.786134 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.786139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:22.786250 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:22.786272 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 08:35:22.786406 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 08:35:22.786536 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"spinlock_wait_cycles":25984,"thread_start_us":779,"threads_started":1}
I20260928 08:35:22.787935 2084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83e5ec45c3f4d019c34844c86c68bf6" candidate_uuid: "59e15008e7a14445b194a5bffe7a41a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfbeccfe994f4fa59e5dc255d061c651" is_pre_election: true
I20260928 08:35:22.788112 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83e5ec45c3f4d019c34844c86c68bf6" candidate_uuid: "59e15008e7a14445b194a5bffe7a41a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56b4bf6cbd7f40699672b12e264d3c3b" is_pre_election: true
I20260928 08:35:22.788486 2084 raft_consensus.cc:2454] T a83e5ec45c3f4d019c34844c86c68bf6 P bfbeccfe994f4fa59e5dc255d061c651 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59e15008e7a14445b194a5bffe7a41a7 in term 0.
I20260928 08:35:22.788755 1949 raft_consensus.cc:2454] T a83e5ec45c3f4d019c34844c86c68bf6 P 56b4bf6cbd7f40699672b12e264d3c3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59e15008e7a14445b194a5bffe7a41a7 in term 0.
I20260928 08:35:22.789225 1751 leader_election.cc:302] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59e15008e7a14445b194a5bffe7a41a7, bfbeccfe994f4fa59e5dc255d061c651; no voters:
I20260928 08:35:22.789914 2154 raft_consensus.cc:2781] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:35:22.790078 2154 raft_consensus.cc:482] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:35:22.790198 2154 raft_consensus.cc:3037] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:22.792357 2154 raft_consensus.cc:504] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e15008e7a14445b194a5bffe7a41a7" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 34871 } } peers { permanent_uuid: "56b4bf6cbd7f40699672b12e264d3c3b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35317 } } peers { permanent_uuid: "bfbeccfe994f4fa59e5dc255d061c651" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44851 } }
I20260928 08:35:22.793020 2154 leader_election.cc:288] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7 [CANDIDATE]: Term 1 election: Requested vote from peers 56b4bf6cbd7f40699672b12e264d3c3b (127.15.134.194:35317), bfbeccfe994f4fa59e5dc255d061c651 (127.15.134.195:44851)
I20260928 08:35:22.793512 2084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83e5ec45c3f4d019c34844c86c68bf6" candidate_uuid: "59e15008e7a14445b194a5bffe7a41a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfbeccfe994f4fa59e5dc255d061c651"
I20260928 08:35:22.793512 1949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a83e5ec45c3f4d019c34844c86c68bf6" candidate_uuid: "59e15008e7a14445b194a5bffe7a41a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56b4bf6cbd7f40699672b12e264d3c3b"
I20260928 08:35:22.793732 2084 raft_consensus.cc:3037] T a83e5ec45c3f4d019c34844c86c68bf6 P bfbeccfe994f4fa59e5dc255d061c651 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:22.793732 1949 raft_consensus.cc:3037] T a83e5ec45c3f4d019c34844c86c68bf6 P 56b4bf6cbd7f40699672b12e264d3c3b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:22.797003 2084 raft_consensus.cc:2454] T a83e5ec45c3f4d019c34844c86c68bf6 P bfbeccfe994f4fa59e5dc255d061c651 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59e15008e7a14445b194a5bffe7a41a7 in term 1.
I20260928 08:35:22.797003 1949 raft_consensus.cc:2454] T a83e5ec45c3f4d019c34844c86c68bf6 P 56b4bf6cbd7f40699672b12e264d3c3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59e15008e7a14445b194a5bffe7a41a7 in term 1.
I20260928 08:35:22.797433 1748 leader_election.cc:302] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56b4bf6cbd7f40699672b12e264d3c3b, 59e15008e7a14445b194a5bffe7a41a7; no voters:
I20260928 08:35:22.797750 2154 raft_consensus.cc:2781] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:35:22.798332 2154 raft_consensus.cc:686] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7 [term 1 LEADER]: Becoming Leader. State: Replica: 59e15008e7a14445b194a5bffe7a41a7, State: Running, Role: LEADER
I20260928 08:35:22.798820 2154 consensus_queue.cc:237] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e15008e7a14445b194a5bffe7a41a7" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 34871 } } peers { permanent_uuid: "56b4bf6cbd7f40699672b12e264d3c3b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35317 } } peers { permanent_uuid: "bfbeccfe994f4fa59e5dc255d061c651" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44851 } }
I20260928 08:35:22.803957 1670 catalog_manager.cc:5495] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59e15008e7a14445b194a5bffe7a41a7 (127.15.134.193). New cstate: current_term: 1 leader_uuid: "59e15008e7a14445b194a5bffe7a41a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59e15008e7a14445b194a5bffe7a41a7" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 34871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "56b4bf6cbd7f40699672b12e264d3c3b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfbeccfe994f4fa59e5dc255d061c651" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44851 } health_report { overall_health: UNKNOWN } } }
I20260928 08:35:22.822562 2143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:22.817000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43594 (local address 127.15.134.193:34871)
0928 08:35:22.817208 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:22.817213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:22.817230 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:35:22.817326 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:22.817330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:22.817409 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:22.817544 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:35:22.817927 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.818721 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.820212 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.820759 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.820902 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:22.821434 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:35:22.821947 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:35:22.822092 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:22.822290 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 08:35:22.822335 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:35:22.831398 2084 raft_consensus.cc:1261] T a83e5ec45c3f4d019c34844c86c68bf6 P bfbeccfe994f4fa59e5dc255d061c651 [term 1 FOLLOWER]: Refusing update from remote peer 59e15008e7a14445b194a5bffe7a41a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:35:22.831406 1949 raft_consensus.cc:1261] T a83e5ec45c3f4d019c34844c86c68bf6 P 56b4bf6cbd7f40699672b12e264d3c3b [term 1 FOLLOWER]: Refusing update from remote peer 59e15008e7a14445b194a5bffe7a41a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:35:22.832484 2159 consensus_queue.cc:1035] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfbeccfe994f4fa59e5dc255d061c651" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:35:22.832912 2154 consensus_queue.cc:1035] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56b4bf6cbd7f40699672b12e264d3c3b" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 35317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 08:35:22.837085 1996 tablet.cc:2369] T a83e5ec45c3f4d019c34844c86c68bf6 P 56b4bf6cbd7f40699672b12e264d3c3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:35:22.845933 2163 mvcc.cc:205] Tried to move back new op lower bound from 7334234205505363968 to 7334234205387808768. Current Snapshot: MvccSnapshot[applied={T|T < 7334234205505363968}]
I20260928 08:35:22.849252 2164 mvcc.cc:205] Tried to move back new op lower bound from 7334234205505363968 to 7334234205387808768. Current Snapshot: MvccSnapshot[applied={T|T < 7334234205505363968}]
I20260928 08:35:22.861166 1671 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:35:22.861459 1671 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:35:22.865520 1671 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3ec18e7d4f2144a298c976e95f7d2bd6: Sending DeleteTablet for 3 replicas of tablet a83e5ec45c3f4d019c34844c86c68bf6
I20260928 08:35:22.866846 1795 tablet_service.cc:1514] Processing DeleteTablet for tablet a83e5ec45c3f4d019c34844c86c68bf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:35:22 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:43578
I20260928 08:35:22.866938 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet a83e5ec45c3f4d019c34844c86c68bf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:35:22 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:35678
I20260928 08:35:22.868291 2064 tablet_service.cc:1514] Processing DeleteTablet for tablet a83e5ec45c3f4d019c34844c86c68bf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:35:22 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:51032
I20260928 08:35:22.869849 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 1732
I20260928 08:35:22.872469 2173 tablet_replica.cc:330] T a83e5ec45c3f4d019c34844c86c68bf6 P bfbeccfe994f4fa59e5dc255d061c651: stopping tablet replica
I20260928 08:35:22.872964 2173 raft_consensus.cc:2229] T a83e5ec45c3f4d019c34844c86c68bf6 P bfbeccfe994f4fa59e5dc255d061c651 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:35:22.873445 2173 raft_consensus.cc:2258] T a83e5ec45c3f4d019c34844c86c68bf6 P bfbeccfe994f4fa59e5dc255d061c651 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:35:22.887324 2173 ts_tablet_manager.cc:1905] T a83e5ec45c3f4d019c34844c86c68bf6 P bfbeccfe994f4fa59e5dc255d061c651: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:35:22.895198 2172 tablet_replica.cc:330] T a83e5ec45c3f4d019c34844c86c68bf6 P 56b4bf6cbd7f40699672b12e264d3c3b: stopping tablet replica
I20260928 08:35:22.895205 2173 ts_tablet_manager.cc:1918] T a83e5ec45c3f4d019c34844c86c68bf6 P bfbeccfe994f4fa59e5dc255d061c651: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:35:22.895370 2173 log.cc:1196] T a83e5ec45c3f4d019c34844c86c68bf6 P bfbeccfe994f4fa59e5dc255d061c651: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/a83e5ec45c3f4d019c34844c86c68bf6
I20260928 08:35:22.895890 2173 ts_tablet_manager.cc:1939] T a83e5ec45c3f4d019c34844c86c68bf6 P bfbeccfe994f4fa59e5dc255d061c651: Deleting consensus metadata
I20260928 08:35:22.896098 2172 raft_consensus.cc:2229] T a83e5ec45c3f4d019c34844c86c68bf6 P 56b4bf6cbd7f40699672b12e264d3c3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:35:22.896726 2172 raft_consensus.cc:2258] T a83e5ec45c3f4d019c34844c86c68bf6 P 56b4bf6cbd7f40699672b12e264d3c3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:35:22.897333 2171 tablet_replica.cc:330] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7: stopping tablet replica
I20260928 08:35:22.898095 2171 raft_consensus.cc:2229] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:35:22.898787 2171 raft_consensus.cc:2258] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 28 08:35:22 dist-test-slave-qp2n krb5kdc[1626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.193@KRBTEST.COM
I20260928 08:35:22.900794 1658 catalog_manager.cc:4841] TS bfbeccfe994f4fa59e5dc255d061c651 (127.15.134.195:44851): tablet a83e5ec45c3f4d019c34844c86c68bf6 (table test-table [id=c3294042b48643b3a7aaedeaf86cc807]) successfully deleted
I20260928 08:35:22.906195 2172 ts_tablet_manager.cc:1905] T a83e5ec45c3f4d019c34844c86c68bf6 P 56b4bf6cbd7f40699672b12e264d3c3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:35:22.908354 2143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:22.871790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43598 (local address 127.15.134.193:34871)
0928 08:35:22.872064 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:22.872073 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:22.887726 (+ 15653us) server_negotiation.cc:407] Connection header received
0928 08:35:22.887922 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:22.887929 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:22.888050 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:22.888192 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:22.890667 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.892252 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.894897 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:22.895999 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:22.896139 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:22.901516 (+ 5377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:22.901529 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:22.901546 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:22.901587 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:22.904715 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:22.905584 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:22.905596 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:22.905603 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:22.905819 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:22.906275 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:22.906280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:22.906283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:22.906740 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:22.907039 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:22.907718 (+ 679us) server_negotiation.cc:299] Negotiation successful
0928 08:35:22.907984 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260928 08:35:22.910595 1855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:22.912334 2172 ts_tablet_manager.cc:1918] T a83e5ec45c3f4d019c34844c86c68bf6 P 56b4bf6cbd7f40699672b12e264d3c3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:35:22.912563 2172 log.cc:1196] T a83e5ec45c3f4d019c34844c86c68bf6 P 56b4bf6cbd7f40699672b12e264d3c3b: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/a83e5ec45c3f4d019c34844c86c68bf6
I20260928 08:35:22.913151 2172 ts_tablet_manager.cc:1939] T a83e5ec45c3f4d019c34844c86c68bf6 P 56b4bf6cbd7f40699672b12e264d3c3b: Deleting consensus metadata
I20260928 08:35:22.915143 1655 catalog_manager.cc:4841] TS 56b4bf6cbd7f40699672b12e264d3c3b (127.15.134.194:35317): tablet a83e5ec45c3f4d019c34844c86c68bf6 (table test-table [id=c3294042b48643b3a7aaedeaf86cc807]) successfully deleted
I20260928 08:35:22.910925 2171 ts_tablet_manager.cc:1905] T a83e5ec45c3f4d019c34844c86c68bf6 P 59e15008e7a14445b194a5bffe7a41a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:35:23.012869 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 1732
I20260928 08:35:23.031929 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 1866
W20260928 08:35:23.032169 1657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.15.134.193:34871 (error 108)
I20260928 08:35:23.034159 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:23.033652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:34871 (local address 127.0.0.1:43602)
0928 08:35:23.033838 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:23.033980 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:34871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 08:35:23.034397 1657 catalog_manager.cc:4584] TS 59e15008e7a14445b194a5bffe7a41a7 (127.15.134.193:34871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a83e5ec45c3f4d019c34844c86c68bf6: Network error: Client connection negotiation failed: client connection to 127.15.134.193:34871: connect: Connection refused (error 111)
Sep 28 08:35:23 dist-test-slave-qp2n krb5kdc[1626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.194@KRBTEST.COM
I20260928 08:35:23.048060 2145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:23.032778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35690 (local address 127.15.134.194:35317)
0928 08:35:23.032963 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:23.032969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:23.033066 (+ 97us) server_negotiation.cc:407] Connection header received
0928 08:35:23.033288 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:23.033292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:23.033361 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:23.033467 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:23.035027 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:23.035758 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:23.037083 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:23.037498 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:23.037627 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:23.041938 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:23.041952 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:23.041967 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:23.042004 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:23.045290 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:23.045963 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:23.045970 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:23.045972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:23.046091 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:23.046489 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:23.046493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:23.046495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:23.046856 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:23.047186 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:23.047491 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 08:35:23.047712 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 08:35:23.049002 1990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:23.088063 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:23.087441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:34871 (local address 127.0.0.1:43616)
0928 08:35:23.087676 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:23.087795 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:34871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 08:35:23.089218 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:23.088816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:34871 (local address 127.0.0.1:43622)
0928 08:35:23.088985 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:23.089070 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:34871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 08:35:23.141510 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 1866
I20260928 08:35:23.150072 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:23.149365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:34871 (local address 127.0.0.1:43638)
0928 08:35:23.149668 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:23.149787 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:34871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260928 08:35:23.151203 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:23.150767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:34871 (local address 127.0.0.1:43648)
0928 08:35:23.150955 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:23.151050 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:34871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 08:35:23.160780 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 2001
Sep 28 08:35:23 dist-test-slave-qp2n krb5kdc[1626](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584520, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.195@KRBTEST.COM
I20260928 08:35:23.176599 2147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:23.161694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51036 (local address 127.15.134.195:44851)
0928 08:35:23.161935 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:23.161942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:23.161964 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:35:23.162103 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:23.162108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:23.162191 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:23.162314 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:23.163570 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:23.164317 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:23.165687 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:23.166103 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:23.166244 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:23.171102 (+ 4858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:23.171118 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:23.171134 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:23.171170 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:23.174069 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:23.174715 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:23.174722 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:23.174725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:23.174877 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:23.175235 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:23.175239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:23.175241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:23.175604 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:23.175817 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:23.176055 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 08:35:23.176232 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:35:23.177285 2124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:23.231702 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:23.231126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:34871 (local address 127.0.0.1:43656)
0928 08:35:23.231373 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:23.231487 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:34871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 08:35:23.233011 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:23.232536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:34871 (local address 127.0.0.1:43670)
0928 08:35:23.232761 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:23.232873 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:34871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260928 08:35:23.270468 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 2001
I20260928 08:35:23.288884 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 1639
I20260928 08:35:23.290067 1701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:23.405620 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 1639
I20260928 08:35:23.405640 2146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:23.405032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:34871 (local address 127.0.0.1:43674)
0928 08:35:23.405370 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:23.405483 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.193:34871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
2026-09-28T08:35:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4363 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T08:35:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:35:23Z Disabled control of system clock
I20260928 08:35:23.487411 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:40099
--webserver_interface=127.15.134.254
--webserver_port=0
--builtin_ntp_servers=127.15.134.212:32957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:40099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:35:23.846225 2183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:23.846923 2183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:23.847096 2183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:23.859506 2183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:35:23.859668 2183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:23.859733 2183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:23.859819 2183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:35:23.859886 2183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:35:23.878372 2183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:32957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:40099
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:40099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:23.880390 2183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:23.882963 2183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:23.895699 2188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:23.895663 2189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:23.895622 2191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:23.896523 2183 server_base.cc:1034] running on GCE node
I20260928 08:35:23.897357 2183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:23.898900 2183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:23.900147 2183 hybrid_clock.cc:648] HybridClock initialized: now 1790584523900102 us; error 38 us; skew 500 ppm
I20260928 08:35:23.900518 2183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:35:23.903057 2183 webserver.cc:460] Webserver started at http://127.15.134.254:33053/ using document root <none> and password file <none>
I20260928 08:35:23.903781 2183 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:23.903925 2183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:23.904250 2183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:23.907284 2183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "ff876a96e7034d39804c9c1224137e1b"
format_stamp: "Formatted at 2026-09-28 08:35:23 on dist-test-slave-qp2n"
server_key: "93d98e60f4a8a95d03bb75ea5606ebb6"
server_key_iv: "4c7afaa19cf2d53f736b3545c5ed7756"
server_key_version: "encryptionkey@0"
I20260928 08:35:23.908155 2183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "ff876a96e7034d39804c9c1224137e1b"
format_stamp: "Formatted at 2026-09-28 08:35:23 on dist-test-slave-qp2n"
server_key: "93d98e60f4a8a95d03bb75ea5606ebb6"
server_key_iv: "4c7afaa19cf2d53f736b3545c5ed7756"
server_key_version: "encryptionkey@0"
I20260928 08:35:23.913537 2183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 08:35:23.917052 2197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:23.918336 2183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:35:23.918526 2183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "ff876a96e7034d39804c9c1224137e1b"
format_stamp: "Formatted at 2026-09-28 08:35:23 on dist-test-slave-qp2n"
server_key: "93d98e60f4a8a95d03bb75ea5606ebb6"
server_key_iv: "4c7afaa19cf2d53f736b3545c5ed7756"
server_key_version: "encryptionkey@0"
I20260928 08:35:23.918745 2183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:23.964026 2183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:23.965091 2183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:23.965411 2183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:23.984808 2183 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:40099
I20260928 08:35:23.984793 2248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:40099 every 8 connection(s)
I20260928 08:35:23.986263 2183 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:35:23.990080 2249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:23.995592 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 2183
I20260928 08:35:23.995805 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:35:23.996289 15899 external_mini_cluster.cc:1428] Setting key b9f3a44ade82837729915fc07c2cc19c
I20260928 08:35:24.001549 2249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff876a96e7034d39804c9c1224137e1b: Bootstrap starting.
I20260928 08:35:24.005102 2249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff876a96e7034d39804c9c1224137e1b: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:24.006398 2249 log.cc:824] T 00000000000000000000000000000000 P ff876a96e7034d39804c9c1224137e1b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:24.009684 2249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff876a96e7034d39804c9c1224137e1b: No bootstrap required, opened a new log
I20260928 08:35:24.011564 2252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:23.998587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56646 (local address 127.15.134.254:40099)
0928 08:35:23.999754 (+ 1167us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:23.999766 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:23.999808 (+ 42us) server_negotiation.cc:407] Connection header received
0928 08:35:24.001596 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:24.001644 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:24.002146 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:24.002481 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:24.003155 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:24.004427 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:24.006418 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:24.007441 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:24.007662 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:24.008953 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:24.008962 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:24.008976 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:24.009601 (+ 625us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:24.009692 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:24.009715 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:24.009899 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:24.010135 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 08:35:24.010499 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":439,"threads_started":1}
I20260928 08:35:24.017613 2249 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff876a96e7034d39804c9c1224137e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff876a96e7034d39804c9c1224137e1b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 40099 } }
I20260928 08:35:24.018024 2249 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff876a96e7034d39804c9c1224137e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:24.018148 2249 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff876a96e7034d39804c9c1224137e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff876a96e7034d39804c9c1224137e1b, State: Initialized, Role: FOLLOWER
I20260928 08:35:24.018707 2249 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff876a96e7034d39804c9c1224137e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff876a96e7034d39804c9c1224137e1b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 40099 } }
I20260928 08:35:24.018955 2249 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff876a96e7034d39804c9c1224137e1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:35:24.019115 2249 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff876a96e7034d39804c9c1224137e1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:35:24.019285 2249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff876a96e7034d39804c9c1224137e1b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:24.021706 2249 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff876a96e7034d39804c9c1224137e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff876a96e7034d39804c9c1224137e1b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 40099 } }
I20260928 08:35:24.022228 2249 leader_election.cc:302] T 00000000000000000000000000000000 P ff876a96e7034d39804c9c1224137e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff876a96e7034d39804c9c1224137e1b; no voters:
I20260928 08:35:24.022763 2249 leader_election.cc:288] T 00000000000000000000000000000000 P ff876a96e7034d39804c9c1224137e1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:35:24.022962 2254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff876a96e7034d39804c9c1224137e1b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:35:24.023782 2254 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff876a96e7034d39804c9c1224137e1b [term 1 LEADER]: Becoming Leader. State: Replica: ff876a96e7034d39804c9c1224137e1b, State: Running, Role: LEADER
I20260928 08:35:24.024382 2254 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff876a96e7034d39804c9c1224137e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff876a96e7034d39804c9c1224137e1b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 40099 } }
I20260928 08:35:24.024987 2249 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff876a96e7034d39804c9c1224137e1b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:35:24.028178 2255 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff876a96e7034d39804c9c1224137e1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff876a96e7034d39804c9c1224137e1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff876a96e7034d39804c9c1224137e1b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 40099 } } }
I20260928 08:35:24.028223 2256 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff876a96e7034d39804c9c1224137e1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff876a96e7034d39804c9c1224137e1b. Latest consensus state: current_term: 1 leader_uuid: "ff876a96e7034d39804c9c1224137e1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff876a96e7034d39804c9c1224137e1b" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 40099 } } }
I20260928 08:35:24.028914 2256 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff876a96e7034d39804c9c1224137e1b [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:24.028915 2255 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff876a96e7034d39804c9c1224137e1b [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:24.032284 2263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:35:24.038084 2263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:35:24.047829 2263 catalog_manager.cc:1347] Generated new cluster ID: 100417227a47461ea27cb43643835ec2
I20260928 08:35:24.047972 2263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:35:24.073143 2263 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:35:24.074481 2263 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:35:24.088994 2263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff876a96e7034d39804c9c1224137e1b: Generated new TSK 0
I20260928 08:35:24.090082 2263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:35:24.098219 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:0
--local_ip_for_outbound_sockets=127.15.134.193
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:40099
--builtin_ntp_servers=127.15.134.212:32957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:35:24.443681 2273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:24.444078 2273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:24.444170 2273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:24.451483 2273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:24.451593 2273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:24.451701 2273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:35:24.460927 2273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:32957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:40099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:24.462234 2273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:24.464116 2273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:24.477229 2278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:24.477301 2279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:24.477329 2281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:24.478545 2273 server_base.cc:1034] running on GCE node
I20260928 08:35:24.479087 2273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:24.480027 2273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:24.481253 2273 hybrid_clock.cc:648] HybridClock initialized: now 1790584524481195 us; error 54 us; skew 500 ppm
I20260928 08:35:24.481575 2273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:35:24.483786 2273 webserver.cc:460] Webserver started at http://127.15.134.193:38269/ using document root <none> and password file <none>
I20260928 08:35:24.484503 2273 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:24.484675 2273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:24.485010 2273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:24.487828 2273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-0/data/instance:
uuid: "af0787994ba242ff9352218ba5bd0391"
format_stamp: "Formatted at 2026-09-28 08:35:24 on dist-test-slave-qp2n"
server_key: "21c266fc6a6bb7bc20580059622d9ce9"
server_key_iv: "a68ff455e246a8d31a117022254ffb60"
server_key_version: "encryptionkey@0"
I20260928 08:35:24.488721 2273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance:
uuid: "af0787994ba242ff9352218ba5bd0391"
format_stamp: "Formatted at 2026-09-28 08:35:24 on dist-test-slave-qp2n"
server_key: "21c266fc6a6bb7bc20580059622d9ce9"
server_key_iv: "a68ff455e246a8d31a117022254ffb60"
server_key_version: "encryptionkey@0"
I20260928 08:35:24.494343 2273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260928 08:35:24.497913 2287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:24.499265 2273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:35:24.499446 2273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "af0787994ba242ff9352218ba5bd0391"
format_stamp: "Formatted at 2026-09-28 08:35:24 on dist-test-slave-qp2n"
server_key: "21c266fc6a6bb7bc20580059622d9ce9"
server_key_iv: "a68ff455e246a8d31a117022254ffb60"
server_key_version: "encryptionkey@0"
I20260928 08:35:24.499679 2273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:24.531272 2273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:24.532181 2273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:24.532492 2273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:24.533557 2273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:35:24.535073 2273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:35:24.535205 2273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:24.535331 2273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:35:24.535424 2273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:24.570289 2273 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:37107
I20260928 08:35:24.570346 2399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:37107 every 8 connection(s)
I20260928 08:35:24.571611 2273 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:35:24.574030 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 2273
I20260928 08:35:24.574213 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance
I20260928 08:35:24.574812 15899 external_mini_cluster.cc:1428] Setting key 0be84cd640419d960a722a734807b6c3
I20260928 08:35:24.577910 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:0
--local_ip_for_outbound_sockets=127.15.134.194
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:40099
--builtin_ntp_servers=127.15.134.212:32957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 08:35:24.586633 2402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:24.574578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:58893 (local address 127.15.134.254:40099)
0928 08:35:24.575106 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:24.575114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:24.576646 (+ 1532us) server_negotiation.cc:407] Connection header received
0928 08:35:24.578659 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:24.578668 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:24.578760 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:24.578891 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:24.580288 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:24.581596 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:24.583326 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:24.584050 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:24.584198 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:24.585662 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:24.585671 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:24.585678 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:24.585708 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:24.585764 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:24.585783 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:24.585881 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:24.586122 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 08:35:24.586243 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":255,"threads_started":1}
I20260928 08:35:24.587996 2403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:24.575291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:40099 (local address 127.15.134.193:58893)
0928 08:35:24.576415 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:24.576472 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:24.578381 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:24.579111 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:24.579119 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:24.579211 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:35:24.579818 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:24.580042 (+ 224us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:24.581773 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:24.581777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:24.583147 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:24.583160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:24.584275 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:24.584280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:24.584498 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:24.585306 (+ 808us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:35:24.585309 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:35:24.585377 (+ 68us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:35:24.585459 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:35:24.585895 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:35:24.585907 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:35:24.585916 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 08:35:24.586073 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 08:35:24.586455 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":314,"threads_started":1}
I20260928 08:35:24.589587 2400 heartbeater.cc:347] Connected to a master server at 127.15.134.254:40099
I20260928 08:35:24.589913 2400 heartbeater.cc:464] Registering TS with master...
I20260928 08:35:24.590865 2400 heartbeater.cc:511] Master 127.15.134.254:40099 requested a full tablet report, sending...
I20260928 08:35:24.593293 2214 ts_manager.cc:194] Registered new tserver with Master: af0787994ba242ff9352218ba5bd0391 (127.15.134.193:37107)
I20260928 08:35:24.594774 2214 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.134.193:58893
W20260928 08:35:24.924966 2404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:24.925386 2404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:24.925565 2404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:24.934681 2404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:24.934861 2404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:24.935067 2404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:35:24.945715 2404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:32957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:40099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:24.947162 2404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:24.949270 2404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:24.964244 2409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:24.964202 2410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:24.964143 2412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:24.965211 2404 server_base.cc:1034] running on GCE node
I20260928 08:35:24.965860 2404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:24.966935 2404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:24.968183 2404 hybrid_clock.cc:648] HybridClock initialized: now 1790584524968128 us; error 51 us; skew 500 ppm
I20260928 08:35:24.968538 2404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:35:24.971243 2404 webserver.cc:460] Webserver started at http://127.15.134.194:36057/ using document root <none> and password file <none>
I20260928 08:35:24.972011 2404 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:24.972146 2404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:24.972463 2404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:24.975574 2404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-1/data/instance:
uuid: "1e5898e9740044b59c496acee659de0d"
format_stamp: "Formatted at 2026-09-28 08:35:24 on dist-test-slave-qp2n"
server_key: "fec84f0601bea246ed6f709bb89b5f89"
server_key_iv: "80f12e43bfd3a30996a6e29b44992d19"
server_key_version: "encryptionkey@0"
I20260928 08:35:24.976382 2404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance:
uuid: "1e5898e9740044b59c496acee659de0d"
format_stamp: "Formatted at 2026-09-28 08:35:24 on dist-test-slave-qp2n"
server_key: "fec84f0601bea246ed6f709bb89b5f89"
server_key_iv: "80f12e43bfd3a30996a6e29b44992d19"
server_key_version: "encryptionkey@0"
I20260928 08:35:24.982064 2404 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 08:35:24.986292 2418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:24.987955 2404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:35:24.988147 2404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "1e5898e9740044b59c496acee659de0d"
format_stamp: "Formatted at 2026-09-28 08:35:24 on dist-test-slave-qp2n"
server_key: "fec84f0601bea246ed6f709bb89b5f89"
server_key_iv: "80f12e43bfd3a30996a6e29b44992d19"
server_key_version: "encryptionkey@0"
I20260928 08:35:24.988387 2404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:25.002530 2404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:25.003482 2404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:25.003789 2404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:25.005172 2404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:35:25.006791 2404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:35:25.006933 2404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:25.007071 2404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:35:25.007160 2404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:25.043393 2404 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:40793
I20260928 08:35:25.043444 2531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:40793 every 8 connection(s)
I20260928 08:35:25.044858 2404 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:35:25.047948 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 2404
I20260928 08:35:25.048130 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance
I20260928 08:35:25.048745 15899 external_mini_cluster.cc:1428] Setting key d4e2652c2b94886cc7455ab192b175a3
I20260928 08:35:25.051782 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:0
--local_ip_for_outbound_sockets=127.15.134.195
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:40099
--builtin_ntp_servers=127.15.134.212:32957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 08:35:25.058537 2402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:25.047557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:54653 (local address 127.15.134.254:40099)
0928 08:35:25.047753 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:25.047758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:25.049772 (+ 2014us) server_negotiation.cc:407] Connection header received
0928 08:35:25.051816 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:25.051821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:25.051908 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:25.052016 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:25.053320 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.054064 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.055656 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.056060 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.056203 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:25.057690 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:25.057696 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:25.057699 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:25.057720 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:25.057752 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:25.057765 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:25.057865 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:25.058106 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 08:35:25.058213 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 08:35:25.059636 2534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:25.048417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:40099 (local address 127.15.134.194:54653)
0928 08:35:25.049586 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:25.049645 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:25.051535 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:25.052373 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:25.052381 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:25.052476 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:35:25.053147 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:25.053165 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.054219 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.054224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:25.055502 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:25.055515 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.056295 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.056302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:25.056548 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:25.057345 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:35:25.057352 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:35:25.057395 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:35:25.057482 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:35:25.057863 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:35:25.057877 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:35:25.057887 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 08:35:25.058085 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 08:35:25.058512 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":385,"threads_started":1}
I20260928 08:35:25.061702 2532 heartbeater.cc:347] Connected to a master server at 127.15.134.254:40099
I20260928 08:35:25.062055 2532 heartbeater.cc:464] Registering TS with master...
I20260928 08:35:25.062896 2532 heartbeater.cc:511] Master 127.15.134.254:40099 requested a full tablet report, sending...
I20260928 08:35:25.064553 2214 ts_manager.cc:194] Registered new tserver with Master: 1e5898e9740044b59c496acee659de0d (127.15.134.194:40793)
I20260928 08:35:25.065898 2214 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.134.194:54653
W20260928 08:35:25.390343 2535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:25.390659 2535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:25.390750 2535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:25.398367 2535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:25.398486 2535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:25.398594 2535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:35:25.407749 2535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:32957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:40099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:25.409027 2535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:25.410718 2535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:25.424698 2540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:25.424638 2541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:25.424587 2543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:25.425717 2535 server_base.cc:1034] running on GCE node
I20260928 08:35:25.426370 2535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:25.427479 2535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:25.428717 2535 hybrid_clock.cc:648] HybridClock initialized: now 1790584525428661 us; error 59 us; skew 500 ppm
I20260928 08:35:25.429119 2535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:35:25.431528 2535 webserver.cc:460] Webserver started at http://127.15.134.195:38601/ using document root <none> and password file <none>
I20260928 08:35:25.432230 2535 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:25.432370 2535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:25.432772 2535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:25.435740 2535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-2/data/instance:
uuid: "f9dc8e34a96e49a1a5781b11c15e8d67"
format_stamp: "Formatted at 2026-09-28 08:35:25 on dist-test-slave-qp2n"
server_key: "8abfd6bb24518a203602f9f635b1e1f8"
server_key_iv: "27a257803600c9001c3541f285cbf011"
server_key_version: "encryptionkey@0"
I20260928 08:35:25.436573 2535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance:
uuid: "f9dc8e34a96e49a1a5781b11c15e8d67"
format_stamp: "Formatted at 2026-09-28 08:35:25 on dist-test-slave-qp2n"
server_key: "8abfd6bb24518a203602f9f635b1e1f8"
server_key_iv: "27a257803600c9001c3541f285cbf011"
server_key_version: "encryptionkey@0"
I20260928 08:35:25.441946 2535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 08:35:25.445430 2549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:25.446771 2535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:35:25.446954 2535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "f9dc8e34a96e49a1a5781b11c15e8d67"
format_stamp: "Formatted at 2026-09-28 08:35:25 on dist-test-slave-qp2n"
server_key: "8abfd6bb24518a203602f9f635b1e1f8"
server_key_iv: "27a257803600c9001c3541f285cbf011"
server_key_version: "encryptionkey@0"
I20260928 08:35:25.447175 2535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:25.468662 2535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:25.469552 2535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:25.469858 2535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:25.471047 2535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:35:25.472689 2535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:35:25.472798 2535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:25.472888 2535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:35:25.472951 2535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:25.506973 2535 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:38305
I20260928 08:35:25.507073 2661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:38305 every 8 connection(s)
I20260928 08:35:25.508391 2535 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:35:25.508881 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 2535
I20260928 08:35:25.509078 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance
I20260928 08:35:25.509723 15899 external_mini_cluster.cc:1428] Setting key a095fc910e7ba00a1c28d3dc1f9bcbd2
I20260928 08:35:25.521629 2402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:25.511265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:48183 (local address 127.15.134.254:40099)
0928 08:35:25.511516 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:25.511521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:25.513303 (+ 1782us) server_negotiation.cc:407] Connection header received
0928 08:35:25.515141 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:25.515147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:25.515237 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:25.515356 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:25.516433 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.517263 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.518835 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.519233 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.519364 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:25.520789 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:25.520794 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:25.520797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:25.520816 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:25.520849 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:25.520861 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:25.520961 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:25.521181 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 08:35:25.521283 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 08:35:25.522874 2664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:25.512022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:40099 (local address 127.15.134.195:48183)
0928 08:35:25.513152 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:25.513190 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:25.514871 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:25.515593 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:25.515601 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:25.515690 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:35:25.516261 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:25.516278 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.517419 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.517431 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:25.518691 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:25.518703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.519471 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.519477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:25.519708 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:25.520433 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:35:25.520435 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:35:25.520481 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:35:25.520559 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:35:25.520959 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:35:25.520972 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:35:25.520981 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 08:35:25.521171 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 08:35:25.521565 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":354,"threads_started":1}
I20260928 08:35:25.524467 2662 heartbeater.cc:347] Connected to a master server at 127.15.134.254:40099
I20260928 08:35:25.524819 2662 heartbeater.cc:464] Registering TS with master...
I20260928 08:35:25.525660 2662 heartbeater.cc:511] Master 127.15.134.254:40099 requested a full tablet report, sending...
I20260928 08:35:25.527033 2213 ts_manager.cc:194] Registered new tserver with Master: f9dc8e34a96e49a1a5781b11c15e8d67 (127.15.134.195:38305)
I20260928 08:35:25.527877 2213 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.134.195:48183
I20260928 08:35:25.537492 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:35:25.548344 2402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:25.541801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56656 (local address 127.15.134.254:40099)
0928 08:35:25.541997 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:25.542028 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:25.542461 (+ 433us) server_negotiation.cc:407] Connection header received
0928 08:35:25.542671 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:25.542683 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:25.542760 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:25.542872 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:25.543411 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.544156 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.545713 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.546207 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.546360 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:25.547591 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:25.547596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:25.547600 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:25.547619 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:25.547649 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:25.547661 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:25.547811 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:25.548011 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 08:35:25.548068 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 08:35:25.552958 2213 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:35:25.559852 2213 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:35:25.598555 2400 heartbeater.cc:503] Master 127.15.134.254:40099 was elected leader, sending a full tablet report...
I20260928 08:35:25.601375 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:25.576293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:38305 (local address 127.0.0.1:48868)
0928 08:35:25.592215 (+ 15922us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:25.592285 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:25.592599 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:25.593659 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:25.593667 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:25.593696 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:25.594216 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:25.594233 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.596199 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.596204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:25.598123 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:25.598142 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.600342 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.600349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:25.600583 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:25.600664 (+ 81us) client_negotiation.cc:769] Sending connection context
0928 08:35:25.600793 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 08:35:25.600994 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15503,"mutex_wait_us":1,"thread_start_us":16516,"threads_started":1}
I20260928 08:35:25.601418 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:25.576569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48868 (local address 127.15.134.195:38305)
0928 08:35:25.592090 (+ 15521us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:25.592098 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:25.592436 (+ 338us) server_negotiation.cc:407] Connection header received
0928 08:35:25.593003 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:25.593012 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:25.593309 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:25.593550 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:25.594443 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.596066 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.598375 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.599786 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.599987 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:25.600207 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:25.600855 (+ 648us) server_negotiation.cc:299] Negotiation successful
0928 08:35:25.601038 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15362,"thread_start_us":15520,"threads_started":1}
I20260928 08:35:25.603160 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:25.576768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:37107 (local address 127.0.0.1:57698)
0928 08:35:25.593085 (+ 16317us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:25.593107 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:25.593287 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:25.595059 (+ 1772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:25.595064 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:25.595090 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:25.595469 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:25.595481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.597540 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.597545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:25.600439 (+ 2894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:25.600459 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.602374 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.602380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:25.602549 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:25.602580 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 08:35:25.602668 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 08:35:25.602808 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16196,"mutex_wait_us":742,"thread_start_us":15514,"threads_started":1}
I20260928 08:35:25.603695 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:25.593581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56464 (local address 127.15.134.194:40793)
0928 08:35:25.594341 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:25.594353 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:25.595044 (+ 691us) server_negotiation.cc:407] Connection header received
0928 08:35:25.595152 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:25.595160 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:25.595353 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:25.595580 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:25.596364 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.597767 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.601143 (+ 3376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.602561 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.602756 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:25.602968 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:25.603085 (+ 117us) server_negotiation.cc:299] Negotiation successful
0928 08:35:25.603252 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":404,"threads_started":1}
I20260928 08:35:25.603956 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:25.576852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57698 (local address 127.15.134.193:37107)
0928 08:35:25.592016 (+ 15164us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:25.592027 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:25.593433 (+ 1406us) server_negotiation.cc:407] Connection header received
0928 08:35:25.594368 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:25.594378 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:25.594601 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:25.594803 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:25.595681 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.597398 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.600691 (+ 3293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.602178 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.602366 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:25.602572 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:25.603449 (+ 877us) server_negotiation.cc:299] Negotiation successful
0928 08:35:25.603610 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14936,"thread_start_us":15181,"threads_started":1}
I20260928 08:35:25.605283 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:25.593301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:40793 (local address 127.0.0.1:56464)
0928 08:35:25.594788 (+ 1487us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:25.594816 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:25.594927 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:25.595808 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:25.595812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:25.595835 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:25.596188 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:25.596198 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.597919 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.597924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:25.600929 (+ 3005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:25.600942 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.602741 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.602747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:25.602911 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:25.602943 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 08:35:25.604879 (+ 1936us) client_negotiation.cc:240] Negotiation successful
0928 08:35:25.605012 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1368,"thread_start_us":1229,"threads_started":1}
I20260928 08:35:25.606396 2597 tablet_service.cc:1467] Processing CreateTablet for tablet d5cdd595c5e44cafb72a9c5b430e8709 (DEFAULT_TABLE table=test-table [id=673975ecf87b4b649596a900da4a7901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:35:25.608045 2597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5cdd595c5e44cafb72a9c5b430e8709. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:25.607815 2335 tablet_service.cc:1467] Processing CreateTablet for tablet d5cdd595c5e44cafb72a9c5b430e8709 (DEFAULT_TABLE table=test-table [id=673975ecf87b4b649596a900da4a7901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:35:25.608219 2467 tablet_service.cc:1467] Processing CreateTablet for tablet d5cdd595c5e44cafb72a9c5b430e8709 (DEFAULT_TABLE table=test-table [id=673975ecf87b4b649596a900da4a7901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:35:25.609287 2335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5cdd595c5e44cafb72a9c5b430e8709. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:25.609555 2467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5cdd595c5e44cafb72a9c5b430e8709. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:25.621443 2680 tablet_bootstrap.cc:492] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d: Bootstrap starting.
I20260928 08:35:25.624123 2681 tablet_bootstrap.cc:492] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391: Bootstrap starting.
I20260928 08:35:25.627794 2680 tablet_bootstrap.cc:654] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:25.627794 2681 tablet_bootstrap.cc:654] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:25.629199 2680 log.cc:824] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:25.629199 2681 log.cc:824] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:25.632817 2682 tablet_bootstrap.cc:492] T d5cdd595c5e44cafb72a9c5b430e8709 P f9dc8e34a96e49a1a5781b11c15e8d67: Bootstrap starting.
I20260928 08:35:25.633373 2680 tablet_bootstrap.cc:492] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d: No bootstrap required, opened a new log
I20260928 08:35:25.633608 2681 tablet_bootstrap.cc:492] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391: No bootstrap required, opened a new log
I20260928 08:35:25.633702 2680 ts_tablet_manager.cc:1397] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260928 08:35:25.633874 2681 ts_tablet_manager.cc:1397] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 08:35:25.636018 2682 tablet_bootstrap.cc:654] T d5cdd595c5e44cafb72a9c5b430e8709 P f9dc8e34a96e49a1a5781b11c15e8d67: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:25.637156 2682 log.cc:824] T d5cdd595c5e44cafb72a9c5b430e8709 P f9dc8e34a96e49a1a5781b11c15e8d67: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:25.640174 2682 tablet_bootstrap.cc:492] T d5cdd595c5e44cafb72a9c5b430e8709 P f9dc8e34a96e49a1a5781b11c15e8d67: No bootstrap required, opened a new log
I20260928 08:35:25.640484 2682 ts_tablet_manager.cc:1397] T d5cdd595c5e44cafb72a9c5b430e8709 P f9dc8e34a96e49a1a5781b11c15e8d67: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260928 08:35:25.645004 2680 raft_consensus.cc:348] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dc8e34a96e49a1a5781b11c15e8d67" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38305 } } peers { permanent_uuid: "af0787994ba242ff9352218ba5bd0391" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 37107 } } peers { permanent_uuid: "1e5898e9740044b59c496acee659de0d" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 40793 } }
I20260928 08:35:25.645004 2681 raft_consensus.cc:348] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dc8e34a96e49a1a5781b11c15e8d67" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38305 } } peers { permanent_uuid: "af0787994ba242ff9352218ba5bd0391" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 37107 } } peers { permanent_uuid: "1e5898e9740044b59c496acee659de0d" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 40793 } }
I20260928 08:35:25.645516 2681 raft_consensus.cc:374] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:25.645515 2680 raft_consensus.cc:374] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:25.645630 2680 raft_consensus.cc:729] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e5898e9740044b59c496acee659de0d, State: Initialized, Role: FOLLOWER
I20260928 08:35:25.645623 2681 raft_consensus.cc:729] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af0787994ba242ff9352218ba5bd0391, State: Initialized, Role: FOLLOWER
I20260928 08:35:25.646337 2681 consensus_queue.cc:260] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dc8e34a96e49a1a5781b11c15e8d67" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38305 } } peers { permanent_uuid: "af0787994ba242ff9352218ba5bd0391" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 37107 } } peers { permanent_uuid: "1e5898e9740044b59c496acee659de0d" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 40793 } }
I20260928 08:35:25.646337 2680 consensus_queue.cc:260] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dc8e34a96e49a1a5781b11c15e8d67" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38305 } } peers { permanent_uuid: "af0787994ba242ff9352218ba5bd0391" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 37107 } } peers { permanent_uuid: "1e5898e9740044b59c496acee659de0d" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 40793 } }
I20260928 08:35:25.648017 2532 heartbeater.cc:503] Master 127.15.134.254:40099 was elected leader, sending a full tablet report...
I20260928 08:35:25.648476 2681 ts_tablet_manager.cc:1428] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260928 08:35:25.648603 2680 ts_tablet_manager.cc:1428] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260928 08:35:25.650717 2682 raft_consensus.cc:348] T d5cdd595c5e44cafb72a9c5b430e8709 P f9dc8e34a96e49a1a5781b11c15e8d67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dc8e34a96e49a1a5781b11c15e8d67" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38305 } } peers { permanent_uuid: "af0787994ba242ff9352218ba5bd0391" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 37107 } } peers { permanent_uuid: "1e5898e9740044b59c496acee659de0d" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 40793 } }
I20260928 08:35:25.651243 2682 raft_consensus.cc:374] T d5cdd595c5e44cafb72a9c5b430e8709 P f9dc8e34a96e49a1a5781b11c15e8d67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:25.651369 2682 raft_consensus.cc:729] T d5cdd595c5e44cafb72a9c5b430e8709 P f9dc8e34a96e49a1a5781b11c15e8d67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9dc8e34a96e49a1a5781b11c15e8d67, State: Initialized, Role: FOLLOWER
I20260928 08:35:25.652092 2682 consensus_queue.cc:260] T d5cdd595c5e44cafb72a9c5b430e8709 P f9dc8e34a96e49a1a5781b11c15e8d67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dc8e34a96e49a1a5781b11c15e8d67" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38305 } } peers { permanent_uuid: "af0787994ba242ff9352218ba5bd0391" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 37107 } } peers { permanent_uuid: "1e5898e9740044b59c496acee659de0d" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 40793 } }
I20260928 08:35:25.653681 2662 heartbeater.cc:503] Master 127.15.134.254:40099 was elected leader, sending a full tablet report...
I20260928 08:35:25.654476 2682 ts_tablet_manager.cc:1428] T d5cdd595c5e44cafb72a9c5b430e8709 P f9dc8e34a96e49a1a5781b11c15e8d67: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260928 08:35:25.747764 2687 raft_consensus.cc:482] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:35:25.748163 2687 raft_consensus.cc:504] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dc8e34a96e49a1a5781b11c15e8d67" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38305 } } peers { permanent_uuid: "af0787994ba242ff9352218ba5bd0391" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 37107 } } peers { permanent_uuid: "1e5898e9740044b59c496acee659de0d" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 40793 } }
I20260928 08:35:25.750044 2687 leader_election.cc:288] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9dc8e34a96e49a1a5781b11c15e8d67 (127.15.134.195:38305), af0787994ba242ff9352218ba5bd0391 (127.15.134.193:37107)
I20260928 08:35:25.756454 2686 raft_consensus.cc:482] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:35:25.756806 2686 raft_consensus.cc:504] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dc8e34a96e49a1a5781b11c15e8d67" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38305 } } peers { permanent_uuid: "af0787994ba242ff9352218ba5bd0391" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 37107 } } peers { permanent_uuid: "1e5898e9740044b59c496acee659de0d" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 40793 } }
I20260928 08:35:25.757699 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:25.750593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:50377 (local address 127.15.134.195:38305)
0928 08:35:25.750863 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:25.750870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:25.751638 (+ 768us) server_negotiation.cc:407] Connection header received
0928 08:35:25.751839 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:25.751852 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:25.751914 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:25.752016 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:25.752741 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.753457 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.755438 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.756377 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.756564 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:25.756657 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:25.757311 (+ 654us) server_negotiation.cc:299] Negotiation successful
0928 08:35:25.757465 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260928 08:35:25.757911 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:25.750351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:38305 (local address 127.15.134.194:50377)
0928 08:35:25.751521 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:25.751537 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:25.751704 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:25.752043 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:25.752048 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:25.752072 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:25.752524 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:25.752534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.753613 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.753619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:25.755243 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:25.755258 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.756762 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.756769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:25.757070 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:25.757156 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 08:35:25.757255 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 08:35:25.757467 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1070,"mutex_wait_us":284,"thread_start_us":394,"threads_started":1}
I20260928 08:35:25.758049 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:25.750351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:37107 (local address 127.15.134.194:53411)
0928 08:35:25.751363 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:25.751385 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:25.751580 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:25.752027 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:25.752032 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:25.752057 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:25.752481 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:25.752495 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.753549 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.753557 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:25.755207 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:25.755220 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.756567 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.756573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:25.757076 (+ 503us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:25.757155 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 08:35:25.757255 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 08:35:25.757467 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":802,"mutex_wait_us":153,"thread_start_us":807,"threads_started":1}
I20260928 08:35:25.759014 2617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5cdd595c5e44cafb72a9c5b430e8709" candidate_uuid: "1e5898e9740044b59c496acee659de0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9dc8e34a96e49a1a5781b11c15e8d67" is_pre_election: true
I20260928 08:35:25.759325 2686 leader_election.cc:288] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f9dc8e34a96e49a1a5781b11c15e8d67 (127.15.134.195:38305), 1e5898e9740044b59c496acee659de0d (127.15.134.194:40793)
I20260928 08:35:25.759701 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:25.750740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:53411 (local address 127.15.134.193:37107)
0928 08:35:25.750983 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:25.750989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:25.751516 (+ 527us) server_negotiation.cc:407] Connection header received
0928 08:35:25.751735 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:25.751743 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:25.751850 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:25.751960 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:25.752669 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.753397 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.755428 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.756346 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.756548 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:25.756658 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:25.759265 (+ 2607us) server_negotiation.cc:299] Negotiation successful
0928 08:35:25.759448 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 08:35:25.759639 2617 raft_consensus.cc:2454] T d5cdd595c5e44cafb72a9c5b430e8709 P f9dc8e34a96e49a1a5781b11c15e8d67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5898e9740044b59c496acee659de0d in term 0.
W20260928 08:35:25.760442 2663 tablet.cc:2369] T d5cdd595c5e44cafb72a9c5b430e8709 P f9dc8e34a96e49a1a5781b11c15e8d67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:35:25.760666 2355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5cdd595c5e44cafb72a9c5b430e8709" candidate_uuid: "1e5898e9740044b59c496acee659de0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0787994ba242ff9352218ba5bd0391" is_pre_election: true
I20260928 08:35:25.761149 2355 raft_consensus.cc:2454] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e5898e9740044b59c496acee659de0d in term 0.
I20260928 08:35:25.762167 2419 leader_election.cc:302] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5898e9740044b59c496acee659de0d, f9dc8e34a96e49a1a5781b11c15e8d67; no voters:
I20260928 08:35:25.763022 2687 raft_consensus.cc:2781] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:35:25.763221 2687 raft_consensus.cc:482] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:35:25.763346 2687 raft_consensus.cc:3037] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:25.766628 2687 raft_consensus.cc:504] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dc8e34a96e49a1a5781b11c15e8d67" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38305 } } peers { permanent_uuid: "af0787994ba242ff9352218ba5bd0391" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 37107 } } peers { permanent_uuid: "1e5898e9740044b59c496acee659de0d" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 40793 } }
I20260928 08:35:25.767835 2617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5cdd595c5e44cafb72a9c5b430e8709" candidate_uuid: "1e5898e9740044b59c496acee659de0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9dc8e34a96e49a1a5781b11c15e8d67"
I20260928 08:35:25.768085 2617 raft_consensus.cc:3037] T d5cdd595c5e44cafb72a9c5b430e8709 P f9dc8e34a96e49a1a5781b11c15e8d67 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:25.768818 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:25.760115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:39339 (local address 127.15.134.195:38305)
0928 08:35:25.761955 (+ 1840us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:25.761960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:25.761977 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:35:25.762039 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:25.762044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:25.762122 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:25.762214 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:25.763135 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.764317 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.766465 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.767529 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.767717 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:25.767793 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:25.768313 (+ 520us) server_negotiation.cc:299] Negotiation successful
0928 08:35:25.768463 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1697}
I20260928 08:35:25.768819 2691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:25.759542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:38305 (local address 127.15.134.193:39339)
0928 08:35:25.760867 (+ 1325us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:25.760898 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:25.761124 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:25.762351 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:25.762358 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:25.762468 (+ 110us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:25.762939 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:25.762955 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.764477 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.764483 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:25.766248 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:25.766260 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.767926 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.767933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:25.768103 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:25.768143 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 08:35:25.768276 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 08:35:25.768417 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1160,"mutex_wait_us":3,"thread_start_us":842,"threads_started":1}
I20260928 08:35:25.769876 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:25.761411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:43717 (local address 127.15.134.194:40793)
0928 08:35:25.761657 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:25.761666 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:25.763016 (+ 1350us) server_negotiation.cc:407] Connection header received
0928 08:35:25.763214 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:25.763219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:25.763349 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:25.763430 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:25.764091 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.765179 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.767664 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.768871 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.769029 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:25.769103 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:25.769504 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 08:35:25.769632 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 08:35:25.769727 2616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5cdd595c5e44cafb72a9c5b430e8709" candidate_uuid: "af0787994ba242ff9352218ba5bd0391" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9dc8e34a96e49a1a5781b11c15e8d67" is_pre_election: true
I20260928 08:35:25.769965 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:25.760941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:40793 (local address 127.15.134.193:43717)
0928 08:35:25.762881 (+ 1940us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:25.762903 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:25.763036 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:25.763643 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:25.763646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:25.763667 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:25.763926 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:25.763933 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.765348 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.765354 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:25.767465 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:25.767480 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.769219 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.769223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:25.769337 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:25.769358 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 08:35:25.769477 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 08:35:25.769609 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1789,"mutex_wait_us":23,"thread_start_us":1582,"threads_started":1}
I20260928 08:35:25.771322 2617 raft_consensus.cc:2454] T d5cdd595c5e44cafb72a9c5b430e8709 P f9dc8e34a96e49a1a5781b11c15e8d67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5898e9740044b59c496acee659de0d in term 1.
I20260928 08:35:25.771773 2419 leader_election.cc:302] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e5898e9740044b59c496acee659de0d, f9dc8e34a96e49a1a5781b11c15e8d67; no voters:
I20260928 08:35:25.772526 2693 raft_consensus.cc:2781] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:35:25.772643 2687 leader_election.cc:288] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d [CANDIDATE]: Term 1 election: Requested vote from peers f9dc8e34a96e49a1a5781b11c15e8d67 (127.15.134.195:38305), af0787994ba242ff9352218ba5bd0391 (127.15.134.193:37107)
I20260928 08:35:25.772377 2487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5cdd595c5e44cafb72a9c5b430e8709" candidate_uuid: "af0787994ba242ff9352218ba5bd0391" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e5898e9740044b59c496acee659de0d" is_pre_election: true
I20260928 08:35:25.772782 2355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5cdd595c5e44cafb72a9c5b430e8709" candidate_uuid: "1e5898e9740044b59c496acee659de0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af0787994ba242ff9352218ba5bd0391"
I20260928 08:35:25.773012 2355 raft_consensus.cc:3037] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:25.773242 2693 raft_consensus.cc:686] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d [term 1 LEADER]: Becoming Leader. State: Replica: 1e5898e9740044b59c496acee659de0d, State: Running, Role: LEADER
I20260928 08:35:25.773875 2693 consensus_queue.cc:237] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dc8e34a96e49a1a5781b11c15e8d67" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38305 } } peers { permanent_uuid: "af0787994ba242ff9352218ba5bd0391" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 37107 } } peers { permanent_uuid: "1e5898e9740044b59c496acee659de0d" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 40793 } }
I20260928 08:35:25.776298 2355 raft_consensus.cc:2454] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e5898e9740044b59c496acee659de0d in term 1.
I20260928 08:35:25.777961 2288 leader_election.cc:302] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af0787994ba242ff9352218ba5bd0391; no voters: 1e5898e9740044b59c496acee659de0d, f9dc8e34a96e49a1a5781b11c15e8d67
I20260928 08:35:25.778452 2686 raft_consensus.cc:2726] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 08:35:25.778836 2213 catalog_manager.cc:5495] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e5898e9740044b59c496acee659de0d (127.15.134.194). New cstate: current_term: 1 leader_uuid: "1e5898e9740044b59c496acee659de0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f9dc8e34a96e49a1a5781b11c15e8d67" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af0787994ba242ff9352218ba5bd0391" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 37107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e5898e9740044b59c496acee659de0d" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 40793 } health_report { overall_health: HEALTHY } } }
W20260928 08:35:25.797075 2533 tablet.cc:2369] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:35:25.812853 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:25.806558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56476 (local address 127.15.134.194:40793)
0928 08:35:25.806808 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:25.806813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:25.806831 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:35:25.807061 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:25.807065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:25.807147 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:25.807271 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:35:25.807611 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.808458 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.810216 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.810699 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.810815 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:25.811512 (+ 697us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:35:25.812030 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:35:25.812174 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:25.812422 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 08:35:25.812503 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 08:35:25.821587 2617 raft_consensus.cc:1261] T d5cdd595c5e44cafb72a9c5b430e8709 P f9dc8e34a96e49a1a5781b11c15e8d67 [term 1 FOLLOWER]: Refusing update from remote peer 1e5898e9740044b59c496acee659de0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:35:25.821635 2355 raft_consensus.cc:1261] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391 [term 1 FOLLOWER]: Refusing update from remote peer 1e5898e9740044b59c496acee659de0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:35:25.822701 2693 consensus_queue.cc:1035] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af0787994ba242ff9352218ba5bd0391" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 37107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:35:25.823071 2694 consensus_queue.cc:1035] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9dc8e34a96e49a1a5781b11c15e8d67" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 08:35:25.824119 2401 tablet.cc:2369] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:35:25.835172 2697 mvcc.cc:205] Tried to move back new op lower bound from 7334234217753456640 to 7334234217574854656. Current Snapshot: MvccSnapshot[applied={T|T < 7334234217753456640}]
I20260928 08:35:25.836663 2699 mvcc.cc:205] Tried to move back new op lower bound from 7334234217753456640 to 7334234217574854656. Current Snapshot: MvccSnapshot[applied={T|T < 7334234217753456640}]
I20260928 08:35:25.849275 2213 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:35:25.849588 2213 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:35:25.853492 2213 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff876a96e7034d39804c9c1224137e1b: Sending DeleteTablet for 3 replicas of tablet d5cdd595c5e44cafb72a9c5b430e8709
I20260928 08:35:25.854823 2335 tablet_service.cc:1514] Processing DeleteTablet for tablet d5cdd595c5e44cafb72a9c5b430e8709 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:35:25 UTC) from {username='slave'} at 127.0.0.1:57698
I20260928 08:35:25.854837 2597 tablet_service.cc:1514] Processing DeleteTablet for tablet d5cdd595c5e44cafb72a9c5b430e8709 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:35:25 UTC) from {username='slave'} at 127.0.0.1:48868
I20260928 08:35:25.855799 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 2273
I20260928 08:35:25.856827 2467 tablet_service.cc:1514] Processing DeleteTablet for tablet d5cdd595c5e44cafb72a9c5b430e8709 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:35:25 UTC) from {username='slave'} at 127.0.0.1:56464
I20260928 08:35:25.865957 2707 tablet_replica.cc:330] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391: stopping tablet replica
I20260928 08:35:25.866568 2707 raft_consensus.cc:2229] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:35:25.867179 2707 raft_consensus.cc:2258] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:35:25.869971 2707 ts_tablet_manager.cc:1905] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:35:25.872742 2706 tablet_replica.cc:330] T d5cdd595c5e44cafb72a9c5b430e8709 P f9dc8e34a96e49a1a5781b11c15e8d67: stopping tablet replica
I20260928 08:35:25.873284 2706 raft_consensus.cc:2229] T d5cdd595c5e44cafb72a9c5b430e8709 P f9dc8e34a96e49a1a5781b11c15e8d67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:35:25.873916 2706 raft_consensus.cc:2258] T d5cdd595c5e44cafb72a9c5b430e8709 P f9dc8e34a96e49a1a5781b11c15e8d67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:35:25.876365 2706 ts_tablet_manager.cc:1905] T d5cdd595c5e44cafb72a9c5b430e8709 P f9dc8e34a96e49a1a5781b11c15e8d67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:35:25.877990 2707 ts_tablet_manager.cc:1918] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:35:25.878221 2707 log.cc:1196] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-0/wal/wals/d5cdd595c5e44cafb72a9c5b430e8709
I20260928 08:35:25.878911 2707 ts_tablet_manager.cc:1939] T d5cdd595c5e44cafb72a9c5b430e8709 P af0787994ba242ff9352218ba5bd0391: Deleting consensus metadata
I20260928 08:35:25.881064 2201 catalog_manager.cc:4841] TS af0787994ba242ff9352218ba5bd0391 (127.15.134.193:37107): tablet d5cdd595c5e44cafb72a9c5b430e8709 (table test-table [id=673975ecf87b4b649596a900da4a7901]) successfully deleted
I20260928 08:35:25.883641 2708 tablet_replica.cc:330] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d: stopping tablet replica
I20260928 08:35:25.884099 2708 raft_consensus.cc:2229] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:35:25.884505 2706 ts_tablet_manager.cc:1918] T d5cdd595c5e44cafb72a9c5b430e8709 P f9dc8e34a96e49a1a5781b11c15e8d67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:35:25.884683 2706 log.cc:1196] T d5cdd595c5e44cafb72a9c5b430e8709 P f9dc8e34a96e49a1a5781b11c15e8d67: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/d5cdd595c5e44cafb72a9c5b430e8709
I20260928 08:35:25.884868 2708 raft_consensus.cc:2258] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:35:25.885214 2706 ts_tablet_manager.cc:1939] T d5cdd595c5e44cafb72a9c5b430e8709 P f9dc8e34a96e49a1a5781b11c15e8d67: Deleting consensus metadata
I20260928 08:35:25.887104 2198 catalog_manager.cc:4841] TS f9dc8e34a96e49a1a5781b11c15e8d67 (127.15.134.195:38305): tablet d5cdd595c5e44cafb72a9c5b430e8709 (table test-table [id=673975ecf87b4b649596a900da4a7901]) successfully deleted
I20260928 08:35:25.887248 2708 ts_tablet_manager.cc:1905] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:35:25.891263 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:25.857818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57712 (local address 127.15.134.193:37107)
0928 08:35:25.858002 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:25.858008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:25.882672 (+ 24664us) server_negotiation.cc:407] Connection header received
0928 08:35:25.882878 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:25.882885 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:25.883007 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:25.883117 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:25.883867 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.885060 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.887113 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:25.887750 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:25.887849 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:25.889445 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:25.889457 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:25.889475 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:25.890306 (+ 831us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:25.890369 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:25.890404 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:25.890520 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:25.890761 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 08:35:25.890892 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:35:25.892091 2395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:25.895938 2708 ts_tablet_manager.cc:1918] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:35:25.896184 2708 log.cc:1196] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/d5cdd595c5e44cafb72a9c5b430e8709
I20260928 08:35:25.896934 2708 ts_tablet_manager.cc:1939] T d5cdd595c5e44cafb72a9c5b430e8709 P 1e5898e9740044b59c496acee659de0d: Deleting consensus metadata
I20260928 08:35:25.900002 2198 catalog_manager.cc:4841] TS 1e5898e9740044b59c496acee659de0d (127.15.134.194:40793): tablet d5cdd595c5e44cafb72a9c5b430e8709 (table test-table [id=673975ecf87b4b649596a900da4a7901]) successfully deleted
I20260928 08:35:26.022038 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 2273
I20260928 08:35:26.043644 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 2404
I20260928 08:35:26.052253 2679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:26.044715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56480 (local address 127.15.134.194:40793)
0928 08:35:26.044925 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:26.044931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:26.044954 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:35:26.045150 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:26.045161 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:26.045275 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:26.045407 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:26.045862 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:26.046675 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:26.048083 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:26.048807 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:26.048968 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:26.050292 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:26.050307 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:26.050325 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:26.051062 (+ 737us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:26.051152 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:26.051179 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:26.051307 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:26.051821 (+ 514us) server_negotiation.cc:299] Negotiation successful
0928 08:35:26.051963 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 08:35:26.053239 2527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:26.170444 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 2404
I20260928 08:35:26.189864 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 2535
I20260928 08:35:26.196973 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:26.190703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48884 (local address 127.15.134.195:38305)
0928 08:35:26.190910 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:26.190915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:26.190955 (+ 40us) server_negotiation.cc:407] Connection header received
0928 08:35:26.191129 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:26.191133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:26.191218 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:26.191307 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:26.191673 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:26.192419 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:26.193810 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:26.194223 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:26.194323 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:26.195485 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:26.195493 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:26.195509 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:26.196113 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:26.196160 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:26.196179 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:26.196266 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:26.196440 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 08:35:26.196522 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 08:35:26.197680 2657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:26.286458 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 2535
I20260928 08:35:26.304548 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 2183
I20260928 08:35:26.305696 2244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:26.423578 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 2183
2026-09-28T08:35:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3018 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:35:26 dist-test-slave-qp2n krb5kdc[2716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:35:26 dist-test-slave-qp2n krb5kdc[2716](info): set up 2 sockets
Sep 28 08:35:26 dist-test-slave-qp2n krb5kdc[2716](info): commencing operation
krb5kdc: starting...
W20260928 08:35:27.554400 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:35:27 dist-test-slave-qp2n krb5kdc[2716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:35:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:35:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:35:27.827595 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:32793
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:40729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:32793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:35:28.163565 2729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:28.163925 2729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:28.164099 2729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:28.171434 2729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:35:28.171589 2729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:28.171700 2729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:28.171808 2729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:35:28.171895 2729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:35:28.182090 2729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:40729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:32793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:32793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:28.183497 2729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:28.184856 2729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:28.196924 2734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:28.196961 2737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:28.196983 2735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:28.197958 2729 server_base.cc:1034] running on GCE node
I20260928 08:35:28.198717 2729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:28.200017 2729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:28.201210 2729 hybrid_clock.cc:648] HybridClock initialized: now 1790584528201173 us; error 33 us; skew 500 ppm
I20260928 08:35:28.201560 2729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:35:28 dist-test-slave-qp2n krb5kdc[2716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584528, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:35:28.209584 2729 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:35:28.211277 2729 webserver.cc:460] Webserver started at http://127.15.134.254:44233/ using document root <none> and password file <none>
I20260928 08:35:28.212065 2729 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:28.212211 2729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:28.212570 2729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:28.215945 2729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "691984ee93e5487bba2d807a186d66d9"
format_stamp: "Formatted at 2026-09-28 08:35:28 on dist-test-slave-qp2n"
server_key: "274b58b0f8362b4c429d7a428cf264b3"
server_key_iv: "9b5a1cd0f1a57779ff8ecb2c20f1a7bc"
server_key_version: "encryptionkey@0"
I20260928 08:35:28.216928 2729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "691984ee93e5487bba2d807a186d66d9"
format_stamp: "Formatted at 2026-09-28 08:35:28 on dist-test-slave-qp2n"
server_key: "274b58b0f8362b4c429d7a428cf264b3"
server_key_iv: "9b5a1cd0f1a57779ff8ecb2c20f1a7bc"
server_key_version: "encryptionkey@0"
I20260928 08:35:28.222858 2729 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 08:35:28.226876 2744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:28.228379 2729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:35:28.228556 2729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "691984ee93e5487bba2d807a186d66d9"
format_stamp: "Formatted at 2026-09-28 08:35:28 on dist-test-slave-qp2n"
server_key: "274b58b0f8362b4c429d7a428cf264b3"
server_key_iv: "9b5a1cd0f1a57779ff8ecb2c20f1a7bc"
server_key_version: "encryptionkey@0"
I20260928 08:35:28.228832 2729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:28.242102 2729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:28.246490 2729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:28.246832 2729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:28.266512 2729 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:32793
I20260928 08:35:28.266552 2795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:32793 every 8 connection(s)
I20260928 08:35:28.268232 2729 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:35:28.272584 2796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:28.273481 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 2729
I20260928 08:35:28.273700 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:35:28.274282 15899 external_mini_cluster.cc:1428] Setting key 0d61729ad21c016668b75068a6d84e99
I20260928 08:35:28.287734 2796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691984ee93e5487bba2d807a186d66d9: Bootstrap starting.
Sep 28 08:35:28 dist-test-slave-qp2n krb5kdc[2716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:35:28.291496 2796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 691984ee93e5487bba2d807a186d66d9: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:28.292598 2796 log.cc:824] T 00000000000000000000000000000000 P 691984ee93e5487bba2d807a186d66d9: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:28.295650 2796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 691984ee93e5487bba2d807a186d66d9: No bootstrap required, opened a new log
I20260928 08:35:28.300127 2799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:28.276481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47198 (local address 127.15.134.254:32793)
0928 08:35:28.277431 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:28.277444 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:28.277489 (+ 45us) server_negotiation.cc:407] Connection header received
0928 08:35:28.279508 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:28.279537 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:28.279960 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:28.280312 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:28.282134 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:28.283445 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:28.285352 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:28.286066 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:28.286204 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:28.291425 (+ 5221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:28.291434 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:28.291450 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:28.291509 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:28.295031 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:28.295944 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:28.295953 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:28.295962 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:28.296202 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:28.296659 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:28.296664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:28.296667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:28.297381 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:28.297709 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:28.298125 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 08:35:28.298611 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":425,"threads_started":1}
I20260928 08:35:28.304941 2796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 691984ee93e5487bba2d807a186d66d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691984ee93e5487bba2d807a186d66d9" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 32793 } }
I20260928 08:35:28.305294 2796 raft_consensus.cc:374] T 00000000000000000000000000000000 P 691984ee93e5487bba2d807a186d66d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:28.305378 2796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 691984ee93e5487bba2d807a186d66d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 691984ee93e5487bba2d807a186d66d9, State: Initialized, Role: FOLLOWER
I20260928 08:35:28.305930 2796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 691984ee93e5487bba2d807a186d66d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691984ee93e5487bba2d807a186d66d9" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 32793 } }
I20260928 08:35:28.306188 2796 raft_consensus.cc:388] T 00000000000000000000000000000000 P 691984ee93e5487bba2d807a186d66d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:35:28.306341 2796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 691984ee93e5487bba2d807a186d66d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:35:28.306523 2796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 691984ee93e5487bba2d807a186d66d9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:28.308809 2796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 691984ee93e5487bba2d807a186d66d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691984ee93e5487bba2d807a186d66d9" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 32793 } }
I20260928 08:35:28.309319 2796 leader_election.cc:302] T 00000000000000000000000000000000 P 691984ee93e5487bba2d807a186d66d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 691984ee93e5487bba2d807a186d66d9; no voters:
I20260928 08:35:28.309934 2796 leader_election.cc:288] T 00000000000000000000000000000000 P 691984ee93e5487bba2d807a186d66d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:35:28.310089 2801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 691984ee93e5487bba2d807a186d66d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:35:28.310840 2801 raft_consensus.cc:686] T 00000000000000000000000000000000 P 691984ee93e5487bba2d807a186d66d9 [term 1 LEADER]: Becoming Leader. State: Replica: 691984ee93e5487bba2d807a186d66d9, State: Running, Role: LEADER
I20260928 08:35:28.311524 2801 consensus_queue.cc:237] T 00000000000000000000000000000000 P 691984ee93e5487bba2d807a186d66d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691984ee93e5487bba2d807a186d66d9" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 32793 } }
I20260928 08:35:28.312408 2796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 691984ee93e5487bba2d807a186d66d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:35:28.314949 2802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691984ee93e5487bba2d807a186d66d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "691984ee93e5487bba2d807a186d66d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691984ee93e5487bba2d807a186d66d9" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 32793 } } }
I20260928 08:35:28.314958 2803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 691984ee93e5487bba2d807a186d66d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 691984ee93e5487bba2d807a186d66d9. Latest consensus state: current_term: 1 leader_uuid: "691984ee93e5487bba2d807a186d66d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "691984ee93e5487bba2d807a186d66d9" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 32793 } } }
I20260928 08:35:28.315510 2803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691984ee93e5487bba2d807a186d66d9 [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:28.315510 2802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 691984ee93e5487bba2d807a186d66d9 [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:28.318307 2809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:35:28.325023 2809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:35:28.335183 2809 catalog_manager.cc:1347] Generated new cluster ID: 3f664632054c4be68e26bc5a779fb1d7
I20260928 08:35:28.335317 2809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:35:28.361022 2809 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:35:28.362452 2809 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:35:28.369993 2809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 691984ee93e5487bba2d807a186d66d9: Generated new TSK 0
I20260928 08:35:28.370949 2809 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.15.134.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
Entry for principal kudu/127.15.134.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab.
I20260928 08:35:28.436187 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:0
--local_ip_for_outbound_sockets=127.15.134.193
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:32793
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:40729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:35:28.783375 2822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:28.783667 2822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:28.783771 2822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:28.791213 2822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:28.791337 2822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:28.791460 2822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:35:28.801532 2822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:40729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.193.keytab
--principal=kudu/127.15.134.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:32793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:28.802810 2822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:28.804108 2822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:28.818500 2827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:28.818470 2828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:28.818698 2830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:28.819276 2822 server_base.cc:1034] running on GCE node
I20260928 08:35:28.819921 2822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:28.820973 2822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:28.822211 2822 hybrid_clock.cc:648] HybridClock initialized: now 1790584528822157 us; error 63 us; skew 500 ppm
I20260928 08:35:28.822556 2822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:35:28 dist-test-slave-qp2n krb5kdc[2716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584528, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:35:28.825870 2822 init.cc:377] Logged in from keytab as kudu/127.15.134.193@KRBTEST.COM (short username kudu)
I20260928 08:35:28.827358 2822 webserver.cc:460] Webserver started at http://127.15.134.193:41139/ using document root <none> and password file <none>
I20260928 08:35:28.828055 2822 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:28.828202 2822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:28.828547 2822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:28.831590 2822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-0/data/instance:
uuid: "70e34638302c4682bf6f0c5b04bd2f48"
format_stamp: "Formatted at 2026-09-28 08:35:28 on dist-test-slave-qp2n"
server_key: "867c20e093a5bc18758a2134c89348ba"
server_key_iv: "874d39dec504fb56b67b59ea36b596ea"
server_key_version: "encryptionkey@0"
I20260928 08:35:28.832367 2822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance:
uuid: "70e34638302c4682bf6f0c5b04bd2f48"
format_stamp: "Formatted at 2026-09-28 08:35:28 on dist-test-slave-qp2n"
server_key: "867c20e093a5bc18758a2134c89348ba"
server_key_iv: "874d39dec504fb56b67b59ea36b596ea"
server_key_version: "encryptionkey@0"
I20260928 08:35:28.837764 2822 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 08:35:28.841284 2837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:28.842763 2822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:35:28.842909 2822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "70e34638302c4682bf6f0c5b04bd2f48"
format_stamp: "Formatted at 2026-09-28 08:35:28 on dist-test-slave-qp2n"
server_key: "867c20e093a5bc18758a2134c89348ba"
server_key_iv: "874d39dec504fb56b67b59ea36b596ea"
server_key_version: "encryptionkey@0"
I20260928 08:35:28.843082 2822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:28.860941 2822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:28.865546 2822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:28.865805 2822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:28.866760 2822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:35:28.868193 2822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:35:28.868269 2822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:28.868371 2822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:35:28.868417 2822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:28.902084 2822 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:33173
I20260928 08:35:28.902122 2950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:33173 every 8 connection(s)
I20260928 08:35:28.903582 2822 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:35:28.913396 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 2822
I20260928 08:35:28.913638 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance
I20260928 08:35:28.914106 15899 external_mini_cluster.cc:1428] Setting key ac560acab98f96325fa00b1ee2b96290
Sep 28 08:35:28 dist-test-slave-qp2n krb5kdc[2716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584528, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.193@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:35:28.925138 2953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:28.906199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:50619 (local address 127.15.134.254:32793)
0928 08:35:28.906670 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:28.906676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:28.907688 (+ 1012us) server_negotiation.cc:407] Connection header received
0928 08:35:28.909632 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:28.909637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:28.909705 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:28.909805 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:28.911374 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:28.912144 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:28.914141 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:28.914611 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:28.914818 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:28.919110 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:28.919116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:28.919119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:28.919151 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:28.921815 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:28.922575 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:28.922580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:28.922591 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:28.922727 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:28.923193 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:28.923202 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:28.923204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:28.923504 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:28.923668 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:28.924477 (+ 809us) server_negotiation.cc:299] Negotiation successful
0928 08:35:28.924752 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":230,"threads_started":1}
I20260928 08:35:28.926241 2954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:28.906812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:32793 (local address 127.15.134.193:50619)
0928 08:35:28.907521 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:28.907560 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:28.909375 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:28.910092 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:28.910100 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:28.910611 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:35:28.911183 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:28.911201 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:28.912322 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:28.912328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:28.913974 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:28.913987 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:28.914994 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:28.915003 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:28.915303 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:28.916070 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:35:28.916124 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:35:28.918893 (+ 2769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:35:28.921972 (+ 3079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:35:28.921982 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:35:28.921998 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:35:28.922442 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:35:28.922854 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:35:28.922861 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:35:28.922865 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:35:28.923064 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:35:28.923633 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:35:28.923643 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:35:28.924090 (+ 447us) client_negotiation.cc:769] Sending connection context
0928 08:35:28.924386 (+ 296us) client_negotiation.cc:240] Negotiation successful
0928 08:35:28.924987 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":274,"threads_started":1}
I20260928 08:35:28.928068 2951 heartbeater.cc:347] Connected to a master server at 127.15.134.254:32793
I20260928 08:35:28.928443 2951 heartbeater.cc:464] Registering TS with master...
I20260928 08:35:28.929476 2951 heartbeater.cc:511] Master 127.15.134.254:32793 requested a full tablet report, sending...
I20260928 08:35:28.932403 2761 ts_manager.cc:194] Registered new tserver with Master: 70e34638302c4682bf6f0c5b04bd2f48 (127.15.134.193:33173)
I20260928 08:35:28.934803 2761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.193@KRBTEST.COM'} at 127.15.134.193:50619
WARNING: no policy specified for kudu/127.15.134.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
Entry for principal kudu/127.15.134.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab.
I20260928 08:35:28.977322 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:0
--local_ip_for_outbound_sockets=127.15.134.194
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:32793
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:40729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:35:29.321321 2957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:29.321645 2957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:29.321818 2957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:29.329774 2957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:29.329957 2957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:29.330132 2957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:35:29.340473 2957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:40729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.194.keytab
--principal=kudu/127.15.134.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:32793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:29.341900 2957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:29.343317 2957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:29.357146 2962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:29.357103 2965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:29.357179 2963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:29.357611 2957 server_base.cc:1034] running on GCE node
I20260928 08:35:29.358274 2957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:29.359149 2957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:29.360419 2957 hybrid_clock.cc:648] HybridClock initialized: now 1790584529360372 us; error 56 us; skew 500 ppm
I20260928 08:35:29.360800 2957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:35:29 dist-test-slave-qp2n krb5kdc[2716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584529, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:35:29.363986 2957 init.cc:377] Logged in from keytab as kudu/127.15.134.194@KRBTEST.COM (short username kudu)
I20260928 08:35:29.365499 2957 webserver.cc:460] Webserver started at http://127.15.134.194:42923/ using document root <none> and password file <none>
I20260928 08:35:29.366194 2957 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:29.366354 2957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:29.366670 2957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:29.369719 2957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-1/data/instance:
uuid: "ea562b70dd224e9aa7296a6b31ea92a8"
format_stamp: "Formatted at 2026-09-28 08:35:29 on dist-test-slave-qp2n"
server_key: "dc5a9f590b1316c2aee1c3390a70b234"
server_key_iv: "03232145c0a0fe43fb8727f70be16fbf"
server_key_version: "encryptionkey@0"
I20260928 08:35:29.370517 2957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance:
uuid: "ea562b70dd224e9aa7296a6b31ea92a8"
format_stamp: "Formatted at 2026-09-28 08:35:29 on dist-test-slave-qp2n"
server_key: "dc5a9f590b1316c2aee1c3390a70b234"
server_key_iv: "03232145c0a0fe43fb8727f70be16fbf"
server_key_version: "encryptionkey@0"
I20260928 08:35:29.376147 2957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260928 08:35:29.380020 2972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:29.381513 2957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:35:29.381714 2957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "ea562b70dd224e9aa7296a6b31ea92a8"
format_stamp: "Formatted at 2026-09-28 08:35:29 on dist-test-slave-qp2n"
server_key: "dc5a9f590b1316c2aee1c3390a70b234"
server_key_iv: "03232145c0a0fe43fb8727f70be16fbf"
server_key_version: "encryptionkey@0"
I20260928 08:35:29.382028 2957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:29.409116 2957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:29.413919 2957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:29.414176 2957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:29.415169 2957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:35:29.416510 2957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:35:29.416582 2957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:29.416745 2957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:35:29.416795 2957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:29.452355 2957 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:40175
I20260928 08:35:29.452455 3084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:40175 every 8 connection(s)
I20260928 08:35:29.453866 2957 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:35:29.463341 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 2957
I20260928 08:35:29.463557 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance
I20260928 08:35:29.464119 15899 external_mini_cluster.cc:1428] Setting key f670b57321393ce884cbe913205a981e
Sep 28 08:35:29 dist-test-slave-qp2n krb5kdc[2716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584529, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.194@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:35:29.478530 3087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:29.457084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:46671 (local address 127.15.134.254:32793)
0928 08:35:29.457587 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:29.457592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:29.458605 (+ 1013us) server_negotiation.cc:407] Connection header received
0928 08:35:29.460807 (+ 2202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:29.460813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:29.460917 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:29.461049 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:29.462701 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:29.463569 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:29.465738 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:29.466361 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:29.466492 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:29.470928 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:29.470937 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:29.470945 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:29.470992 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:29.474868 (+ 3876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:29.475760 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:29.475769 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:29.475773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:29.475961 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:29.476469 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:29.476475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:29.476479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:29.476864 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:29.477030 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:29.477856 (+ 826us) server_negotiation.cc:299] Negotiation successful
0928 08:35:29.478109 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":258,"threads_started":1}
I20260928 08:35:29.479630 3088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:29.457641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:32793 (local address 127.15.134.194:46671)
0928 08:35:29.458430 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:29.458471 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:29.460438 (+ 1967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:29.461319 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:29.461328 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:29.461912 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:35:29.462513 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:29.462531 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:29.463785 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:29.463790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:29.465589 (+ 1799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:29.465611 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:29.466584 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:29.466589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:29.466827 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:29.467664 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:35:29.467691 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:35:29.470650 (+ 2959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:35:29.475038 (+ 4388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:35:29.475048 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:35:29.475070 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:35:29.475581 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:35:29.476092 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:35:29.476097 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:35:29.476102 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:35:29.476330 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:35:29.477014 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:35:29.477023 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:35:29.477421 (+ 398us) client_negotiation.cc:769] Sending connection context
0928 08:35:29.477717 (+ 296us) client_negotiation.cc:240] Negotiation successful
0928 08:35:29.478335 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":313,"threads_started":1}
I20260928 08:35:29.481293 3085 heartbeater.cc:347] Connected to a master server at 127.15.134.254:32793
I20260928 08:35:29.481673 3085 heartbeater.cc:464] Registering TS with master...
I20260928 08:35:29.482556 3085 heartbeater.cc:511] Master 127.15.134.254:32793 requested a full tablet report, sending...
I20260928 08:35:29.484261 2761 ts_manager.cc:194] Registered new tserver with Master: ea562b70dd224e9aa7296a6b31ea92a8 (127.15.134.194:40175)
I20260928 08:35:29.485237 2761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.194@KRBTEST.COM'} at 127.15.134.194:46671
WARNING: no policy specified for kudu/127.15.134.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
Entry for principal kudu/127.15.134.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab.
I20260928 08:35:29.524718 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:0
--local_ip_for_outbound_sockets=127.15.134.195
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:32793
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:40729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:35:29.875389 3091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:29.875762 3091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:29.875941 3091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:29.883710 3091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:29.883888 3091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:29.884065 3091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:35:29.894270 3091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:40729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.195.keytab
--principal=kudu/127.15.134.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:32793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:29.895592 3091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:29.897004 3091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:29.911321 3096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:29.911437 3097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:29.911414 3099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:29.918200 3091 server_base.cc:1034] running on GCE node
I20260928 08:35:29.918836 3091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:29.919919 3091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:29.921168 3091 hybrid_clock.cc:648] HybridClock initialized: now 1790584529921121 us; error 33 us; skew 500 ppm
I20260928 08:35:29.921546 3091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:35:29 dist-test-slave-qp2n krb5kdc[2716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584529, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:35:29.925418 3091 init.cc:377] Logged in from keytab as kudu/127.15.134.195@KRBTEST.COM (short username kudu)
I20260928 08:35:29.927114 3091 webserver.cc:460] Webserver started at http://127.15.134.195:39471/ using document root <none> and password file <none>
I20260928 08:35:29.927883 3091 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:29.928035 3091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:29.928380 3091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:29.931552 3091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-2/data/instance:
uuid: "11d2d3cd11a54b2ea5ab43d98370a38b"
format_stamp: "Formatted at 2026-09-28 08:35:29 on dist-test-slave-qp2n"
server_key: "082898f7ea5e239d56c89fef90423443"
server_key_iv: "27156b7a8887c6e5c57a69f075842d9b"
server_key_version: "encryptionkey@0"
I20260928 08:35:29.932428 3091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance:
uuid: "11d2d3cd11a54b2ea5ab43d98370a38b"
format_stamp: "Formatted at 2026-09-28 08:35:29 on dist-test-slave-qp2n"
server_key: "082898f7ea5e239d56c89fef90423443"
server_key_iv: "27156b7a8887c6e5c57a69f075842d9b"
server_key_version: "encryptionkey@0"
I20260928 08:35:29.938318 3091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 08:35:29.939348 2951 heartbeater.cc:503] Master 127.15.134.254:32793 was elected leader, sending a full tablet report...
I20260928 08:35:29.942660 3106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:29.944147 3091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:35:29.944300 3091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "11d2d3cd11a54b2ea5ab43d98370a38b"
format_stamp: "Formatted at 2026-09-28 08:35:29 on dist-test-slave-qp2n"
server_key: "082898f7ea5e239d56c89fef90423443"
server_key_iv: "27156b7a8887c6e5c57a69f075842d9b"
server_key_version: "encryptionkey@0"
I20260928 08:35:29.944475 3091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:29.958376 3091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:29.963052 3091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:29.963363 3091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:29.964416 3091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:35:29.965945 3091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:35:29.966028 3091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:29.966121 3091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:35:29.966179 3091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:30.001709 3091 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:38299
I20260928 08:35:30.001760 3218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:38299 every 8 connection(s)
I20260928 08:35:30.003077 3091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:35:30.011485 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 3091
I20260928 08:35:30.011710 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance
I20260928 08:35:30.012308 15899 external_mini_cluster.cc:1428] Setting key 2202b2ddc07409b77ce2b5c5ba681e69
Sep 28 08:35:30 dist-test-slave-qp2n krb5kdc[2716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584529, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.195@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:35:30.025208 3221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:30.005737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:33031 (local address 127.15.134.254:32793)
0928 08:35:30.006225 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:30.006229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:30.007030 (+ 801us) server_negotiation.cc:407] Connection header received
0928 08:35:30.008951 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:30.008957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:30.009065 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:30.009212 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:30.010791 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.011585 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.014117 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.014776 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.014921 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:30.018615 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:30.018621 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:30.018624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:30.018658 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:30.021620 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:30.022615 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:30.022621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:30.022623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:30.022878 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:30.023368 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:30.023372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:30.023374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:30.023680 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:30.023868 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:30.024730 (+ 862us) server_negotiation.cc:299] Negotiation successful
0928 08:35:30.024926 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":262,"threads_started":1}
I20260928 08:35:30.026306 3222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:30.006165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:32793 (local address 127.15.134.195:33031)
0928 08:35:30.006863 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:30.006900 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:30.008602 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:30.009467 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:30.009475 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:30.009998 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:35:30.010604 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:30.010623 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.011784 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.011792 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:30.013927 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:30.013946 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.015005 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.015011 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:30.015217 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:30.016000 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:35:30.016027 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:35:30.018407 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:35:30.021795 (+ 3388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:35:30.021803 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:35:30.021818 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:35:30.022405 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:35:30.023009 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:35:30.023013 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:35:30.023016 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:35:30.023240 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:35:30.023883 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:35:30.023889 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:35:30.024304 (+ 415us) client_negotiation.cc:769] Sending connection context
0928 08:35:30.024568 (+ 264us) client_negotiation.cc:240] Negotiation successful
0928 08:35:30.025078 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":281,"threads_started":1}
I20260928 08:35:30.027820 3219 heartbeater.cc:347] Connected to a master server at 127.15.134.254:32793
I20260928 08:35:30.028144 3219 heartbeater.cc:464] Registering TS with master...
I20260928 08:35:30.028961 3219 heartbeater.cc:511] Master 127.15.134.254:32793 requested a full tablet report, sending...
I20260928 08:35:30.030442 2761 ts_manager.cc:194] Registered new tserver with Master: 11d2d3cd11a54b2ea5ab43d98370a38b (127.15.134.195:38299)
I20260928 08:35:30.031267 2761 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.15.134.195@KRBTEST.COM'} at 127.15.134.195:33031
I20260928 08:35:30.040733 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:35:30.058274 3221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:30.045139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47202 (local address 127.15.134.254:32793)
0928 08:35:30.045334 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:30.045338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:30.045870 (+ 532us) server_negotiation.cc:407] Connection header received
0928 08:35:30.046086 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:30.046095 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:30.046162 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:30.046268 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:30.047744 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.048553 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.049993 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.050399 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.050534 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:30.053026 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:30.053031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:30.053034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:30.053067 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:30.055570 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:30.056199 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:30.056204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:30.056208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:30.056338 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:30.056761 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:30.056767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:30.056771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:30.057104 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:30.057359 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:30.057710 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 08:35:30.057949 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 08:35:30.062289 2761 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:35:30.069190 2761 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:35:30.104050 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:30.082183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43462 (local address 127.15.134.193:33173)
0928 08:35:30.089916 (+ 7733us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:30.089929 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:30.089971 (+ 42us) server_negotiation.cc:407] Connection header received
0928 08:35:30.090104 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:30.090116 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:30.090466 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:30.093018 (+ 2552us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:30.093344 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.095473 (+ 2129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.101079 (+ 5606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.102716 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.102916 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:30.103097 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:30.103504 (+ 407us) server_negotiation.cc:299] Negotiation successful
0928 08:35:30.103691 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7523,"thread_start_us":7357,"threads_started":1}
I20260928 08:35:30.104089 3232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:30.082015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:33173 (local address 127.0.0.1:43462)
0928 08:35:30.082930 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:30.082980 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:30.083155 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:30.092087 (+ 8932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:30.092100 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:30.092149 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:30.092714 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:30.092732 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.098263 (+ 5531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.098268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:30.100864 (+ 2596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:30.100879 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.102958 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.102967 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:30.103202 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:30.103286 (+ 84us) client_negotiation.cc:769] Sending connection context
0928 08:35:30.103442 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 08:35:30.103659 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":889,"threads_started":1}
I20260928 08:35:30.107239 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:30.082458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:40175 (local address 127.0.0.1:47720)
0928 08:35:30.097950 (+ 15492us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:30.097979 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:30.098112 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:30.099123 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:30.099128 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:30.099153 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:30.099640 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:30.099654 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.101518 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.101524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:30.103670 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:30.103684 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.105962 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.105970 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:30.106163 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:30.106196 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 08:35:30.106651 (+ 455us) client_negotiation.cc:240] Negotiation successful
0928 08:35:30.106802 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15330,"mutex_wait_us":1,"thread_start_us":15133,"threads_started":1}
I20260928 08:35:30.107844 3237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:30.083573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42274 (local address 127.15.134.195:38299)
0928 08:35:30.098808 (+ 15235us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:30.098817 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:30.098860 (+ 43us) server_negotiation.cc:407] Connection header received
0928 08:35:30.098967 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:30.098975 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:30.099189 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:30.099395 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:30.100170 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.101515 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.104869 (+ 3354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.106424 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.106631 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:30.106822 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:30.107316 (+ 494us) server_negotiation.cc:299] Negotiation successful
0928 08:35:30.107481 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15017,"thread_start_us":16066,"threads_started":1}
I20260928 08:35:30.108048 3236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:30.083086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47720 (local address 127.15.134.194:40175)
0928 08:35:30.096359 (+ 13273us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:30.096373 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:30.098109 (+ 1736us) server_negotiation.cc:407] Connection header received
0928 08:35:30.098335 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:30.098346 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:30.098644 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:30.098841 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:30.099872 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.101361 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.103849 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.105451 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.105620 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:30.105804 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:30.107555 (+ 1751us) server_negotiation.cc:299] Negotiation successful
0928 08:35:30.107728 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13054,"thread_start_us":13307,"threads_started":1}
I20260928 08:35:30.107864 3234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:30.082328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:38299 (local address 127.0.0.1:42274)
0928 08:35:30.097177 (+ 14849us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:30.097209 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:30.097357 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:30.099394 (+ 2037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:30.099400 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:30.099427 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:30.099967 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:30.099979 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.101750 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.101759 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:30.104635 (+ 2876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:30.104653 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.106985 (+ 2332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.106990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:30.107121 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:30.107145 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 08:35:30.107272 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 08:35:30.107406 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14673,"mutex_wait_us":286,"thread_start_us":14461,"threads_started":1}
I20260928 08:35:30.108875 2885 tablet_service.cc:1467] Processing CreateTablet for tablet 3973adfac4244105b6942b03d9399da7 (DEFAULT_TABLE table=test-table [id=3208e460374a4e61a65cc970c6a3a791]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:35:30.110513 2885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3973adfac4244105b6942b03d9399da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:30.112188 3154 tablet_service.cc:1467] Processing CreateTablet for tablet 3973adfac4244105b6942b03d9399da7 (DEFAULT_TABLE table=test-table [id=3208e460374a4e61a65cc970c6a3a791]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:35:30.112130 3020 tablet_service.cc:1467] Processing CreateTablet for tablet 3973adfac4244105b6942b03d9399da7 (DEFAULT_TABLE table=test-table [id=3208e460374a4e61a65cc970c6a3a791]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:35:30.113912 3020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3973adfac4244105b6942b03d9399da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:30.113948 3154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3973adfac4244105b6942b03d9399da7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:30.123142 3238 tablet_bootstrap.cc:492] T 3973adfac4244105b6942b03d9399da7 P 70e34638302c4682bf6f0c5b04bd2f48: Bootstrap starting.
I20260928 08:35:30.126575 3238 tablet_bootstrap.cc:654] T 3973adfac4244105b6942b03d9399da7 P 70e34638302c4682bf6f0c5b04bd2f48: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:30.128142 3238 log.cc:824] T 3973adfac4244105b6942b03d9399da7 P 70e34638302c4682bf6f0c5b04bd2f48: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:30.133338 3240 tablet_bootstrap.cc:492] T 3973adfac4244105b6942b03d9399da7 P 11d2d3cd11a54b2ea5ab43d98370a38b: Bootstrap starting.
I20260928 08:35:30.139976 3238 tablet_bootstrap.cc:492] T 3973adfac4244105b6942b03d9399da7 P 70e34638302c4682bf6f0c5b04bd2f48: No bootstrap required, opened a new log
I20260928 08:35:30.140256 3240 tablet_bootstrap.cc:654] T 3973adfac4244105b6942b03d9399da7 P 11d2d3cd11a54b2ea5ab43d98370a38b: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:30.140432 3238 ts_tablet_manager.cc:1397] T 3973adfac4244105b6942b03d9399da7 P 70e34638302c4682bf6f0c5b04bd2f48: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.010s
I20260928 08:35:30.141203 3239 tablet_bootstrap.cc:492] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8: Bootstrap starting.
I20260928 08:35:30.144658 3239 tablet_bootstrap.cc:654] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:30.146106 3239 log.cc:824] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:30.146262 3240 log.cc:824] T 3973adfac4244105b6942b03d9399da7 P 11d2d3cd11a54b2ea5ab43d98370a38b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:30.153831 3238 raft_consensus.cc:348] T 3973adfac4244105b6942b03d9399da7 P 70e34638302c4682bf6f0c5b04bd2f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e34638302c4682bf6f0c5b04bd2f48" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33173 } } peers { permanent_uuid: "11d2d3cd11a54b2ea5ab43d98370a38b" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38299 } } peers { permanent_uuid: "ea562b70dd224e9aa7296a6b31ea92a8" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 40175 } }
I20260928 08:35:30.154297 3238 raft_consensus.cc:374] T 3973adfac4244105b6942b03d9399da7 P 70e34638302c4682bf6f0c5b04bd2f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:30.154459 3238 raft_consensus.cc:729] T 3973adfac4244105b6942b03d9399da7 P 70e34638302c4682bf6f0c5b04bd2f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e34638302c4682bf6f0c5b04bd2f48, State: Initialized, Role: FOLLOWER
I20260928 08:35:30.154523 3239 tablet_bootstrap.cc:492] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8: No bootstrap required, opened a new log
I20260928 08:35:30.154897 3239 ts_tablet_manager.cc:1397] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.010s
I20260928 08:35:30.155259 3240 tablet_bootstrap.cc:492] T 3973adfac4244105b6942b03d9399da7 P 11d2d3cd11a54b2ea5ab43d98370a38b: No bootstrap required, opened a new log
I20260928 08:35:30.155243 3238 consensus_queue.cc:260] T 3973adfac4244105b6942b03d9399da7 P 70e34638302c4682bf6f0c5b04bd2f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e34638302c4682bf6f0c5b04bd2f48" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33173 } } peers { permanent_uuid: "11d2d3cd11a54b2ea5ab43d98370a38b" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38299 } } peers { permanent_uuid: "ea562b70dd224e9aa7296a6b31ea92a8" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 40175 } }
I20260928 08:35:30.155557 3240 ts_tablet_manager.cc:1397] T 3973adfac4244105b6942b03d9399da7 P 11d2d3cd11a54b2ea5ab43d98370a38b: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.000s
I20260928 08:35:30.157413 3238 ts_tablet_manager.cc:1428] T 3973adfac4244105b6942b03d9399da7 P 70e34638302c4682bf6f0c5b04bd2f48: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260928 08:35:30.166257 3240 raft_consensus.cc:348] T 3973adfac4244105b6942b03d9399da7 P 11d2d3cd11a54b2ea5ab43d98370a38b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e34638302c4682bf6f0c5b04bd2f48" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33173 } } peers { permanent_uuid: "11d2d3cd11a54b2ea5ab43d98370a38b" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38299 } } peers { permanent_uuid: "ea562b70dd224e9aa7296a6b31ea92a8" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 40175 } }
I20260928 08:35:30.166257 3239 raft_consensus.cc:348] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e34638302c4682bf6f0c5b04bd2f48" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33173 } } peers { permanent_uuid: "11d2d3cd11a54b2ea5ab43d98370a38b" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38299 } } peers { permanent_uuid: "ea562b70dd224e9aa7296a6b31ea92a8" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 40175 } }
I20260928 08:35:30.166762 3240 raft_consensus.cc:374] T 3973adfac4244105b6942b03d9399da7 P 11d2d3cd11a54b2ea5ab43d98370a38b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:30.166762 3239 raft_consensus.cc:374] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:30.166922 3239 raft_consensus.cc:729] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea562b70dd224e9aa7296a6b31ea92a8, State: Initialized, Role: FOLLOWER
I20260928 08:35:30.166927 3240 raft_consensus.cc:729] T 3973adfac4244105b6942b03d9399da7 P 11d2d3cd11a54b2ea5ab43d98370a38b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11d2d3cd11a54b2ea5ab43d98370a38b, State: Initialized, Role: FOLLOWER
I20260928 08:35:30.167610 3240 consensus_queue.cc:260] T 3973adfac4244105b6942b03d9399da7 P 11d2d3cd11a54b2ea5ab43d98370a38b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e34638302c4682bf6f0c5b04bd2f48" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33173 } } peers { permanent_uuid: "11d2d3cd11a54b2ea5ab43d98370a38b" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38299 } } peers { permanent_uuid: "ea562b70dd224e9aa7296a6b31ea92a8" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 40175 } }
I20260928 08:35:30.167608 3239 consensus_queue.cc:260] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e34638302c4682bf6f0c5b04bd2f48" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33173 } } peers { permanent_uuid: "11d2d3cd11a54b2ea5ab43d98370a38b" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38299 } } peers { permanent_uuid: "ea562b70dd224e9aa7296a6b31ea92a8" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 40175 } }
I20260928 08:35:30.168819 3085 heartbeater.cc:503] Master 127.15.134.254:32793 was elected leader, sending a full tablet report...
I20260928 08:35:30.169250 3219 heartbeater.cc:503] Master 127.15.134.254:32793 was elected leader, sending a full tablet report...
I20260928 08:35:30.169646 3239 ts_tablet_manager.cc:1428] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260928 08:35:30.169843 3240 ts_tablet_manager.cc:1428] T 3973adfac4244105b6942b03d9399da7 P 11d2d3cd11a54b2ea5ab43d98370a38b: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260928 08:35:30.206221 3086 tablet.cc:2369] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:35:30.255151 3220 tablet.cc:2369] T 3973adfac4244105b6942b03d9399da7 P 11d2d3cd11a54b2ea5ab43d98370a38b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:35:30.315140 3246 raft_consensus.cc:482] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:35:30.315450 3246 raft_consensus.cc:504] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e34638302c4682bf6f0c5b04bd2f48" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33173 } } peers { permanent_uuid: "11d2d3cd11a54b2ea5ab43d98370a38b" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38299 } } peers { permanent_uuid: "ea562b70dd224e9aa7296a6b31ea92a8" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 40175 } }
I20260928 08:35:30.317195 3246 leader_election.cc:288] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70e34638302c4682bf6f0c5b04bd2f48 (127.15.134.193:33173), 11d2d3cd11a54b2ea5ab43d98370a38b (127.15.134.195:38299)
I20260928 08:35:30.326649 3237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:30.317856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:33237 (local address 127.15.134.195:38299)
0928 08:35:30.318100 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:30.318109 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:30.318513 (+ 404us) server_negotiation.cc:407] Connection header received
0928 08:35:30.318762 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:30.318767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:30.318851 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:30.318953 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:30.319690 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.320907 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.324075 (+ 3168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.325447 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.325587 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:30.325649 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:30.326243 (+ 594us) server_negotiation.cc:299] Negotiation successful
0928 08:35:30.326380 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 08:35:30.326653 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:30.317553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:42451 (local address 127.15.134.193:33173)
0928 08:35:30.317807 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:30.317817 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:30.318840 (+ 1023us) server_negotiation.cc:407] Connection header received
0928 08:35:30.318979 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:30.318984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:30.319055 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:30.319143 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:30.319925 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.321018 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.324075 (+ 3057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.325388 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.325573 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:30.325649 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:30.326243 (+ 594us) server_negotiation.cc:299] Negotiation successful
0928 08:35:30.326380 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 08:35:30.326978 3248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:30.317560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:33173 (local address 127.15.134.194:42451)
0928 08:35:30.318732 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:30.318750 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:30.318853 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:30.319182 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:30.319189 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:30.319213 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:30.319654 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:30.319667 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.321149 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.321154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:30.323843 (+ 2689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:30.323865 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.325728 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.325734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:30.325966 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:30.326081 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 08:35:30.326185 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 08:35:30.326455 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1060,"mutex_wait_us":282,"thread_start_us":829,"threads_started":1}
I20260928 08:35:30.327001 3247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:30.317534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:38299 (local address 127.15.134.194:33237)
0928 08:35:30.318388 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:30.318405 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:30.318607 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:30.318996 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:30.318999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:30.319018 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:30.319508 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:30.319524 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.321059 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.321062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:30.323869 (+ 2807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:30.323878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.325704 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.325716 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:30.325978 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:30.326081 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 08:35:30.326185 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 08:35:30.326455 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":278,"thread_start_us":491,"threads_started":1}
I20260928 08:35:30.328006 2906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3973adfac4244105b6942b03d9399da7" candidate_uuid: "ea562b70dd224e9aa7296a6b31ea92a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70e34638302c4682bf6f0c5b04bd2f48" is_pre_election: true
I20260928 08:35:30.328082 3174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3973adfac4244105b6942b03d9399da7" candidate_uuid: "ea562b70dd224e9aa7296a6b31ea92a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11d2d3cd11a54b2ea5ab43d98370a38b" is_pre_election: true
I20260928 08:35:30.328557 3174 raft_consensus.cc:2454] T 3973adfac4244105b6942b03d9399da7 P 11d2d3cd11a54b2ea5ab43d98370a38b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea562b70dd224e9aa7296a6b31ea92a8 in term 0.
I20260928 08:35:30.328555 2906 raft_consensus.cc:2454] T 3973adfac4244105b6942b03d9399da7 P 70e34638302c4682bf6f0c5b04bd2f48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea562b70dd224e9aa7296a6b31ea92a8 in term 0.
I20260928 08:35:30.329299 2976 leader_election.cc:302] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11d2d3cd11a54b2ea5ab43d98370a38b, ea562b70dd224e9aa7296a6b31ea92a8; no voters:
I20260928 08:35:30.329744 3246 raft_consensus.cc:2781] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:35:30.329905 3246 raft_consensus.cc:482] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:35:30.330024 3246 raft_consensus.cc:3037] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:30.332247 3246 raft_consensus.cc:504] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e34638302c4682bf6f0c5b04bd2f48" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33173 } } peers { permanent_uuid: "11d2d3cd11a54b2ea5ab43d98370a38b" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38299 } } peers { permanent_uuid: "ea562b70dd224e9aa7296a6b31ea92a8" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 40175 } }
I20260928 08:35:30.332981 3246 leader_election.cc:288] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8 [CANDIDATE]: Term 1 election: Requested vote from peers 70e34638302c4682bf6f0c5b04bd2f48 (127.15.134.193:33173), 11d2d3cd11a54b2ea5ab43d98370a38b (127.15.134.195:38299)
I20260928 08:35:30.333542 3174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3973adfac4244105b6942b03d9399da7" candidate_uuid: "ea562b70dd224e9aa7296a6b31ea92a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11d2d3cd11a54b2ea5ab43d98370a38b"
I20260928 08:35:30.333570 2906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3973adfac4244105b6942b03d9399da7" candidate_uuid: "ea562b70dd224e9aa7296a6b31ea92a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70e34638302c4682bf6f0c5b04bd2f48"
I20260928 08:35:30.333832 2906 raft_consensus.cc:3037] T 3973adfac4244105b6942b03d9399da7 P 70e34638302c4682bf6f0c5b04bd2f48 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:30.333904 3174 raft_consensus.cc:3037] T 3973adfac4244105b6942b03d9399da7 P 11d2d3cd11a54b2ea5ab43d98370a38b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:30.337211 3174 raft_consensus.cc:2454] T 3973adfac4244105b6942b03d9399da7 P 11d2d3cd11a54b2ea5ab43d98370a38b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea562b70dd224e9aa7296a6b31ea92a8 in term 1.
I20260928 08:35:30.337219 2906 raft_consensus.cc:2454] T 3973adfac4244105b6942b03d9399da7 P 70e34638302c4682bf6f0c5b04bd2f48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea562b70dd224e9aa7296a6b31ea92a8 in term 1.
I20260928 08:35:30.337724 2974 leader_election.cc:302] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 11d2d3cd11a54b2ea5ab43d98370a38b, 70e34638302c4682bf6f0c5b04bd2f48, ea562b70dd224e9aa7296a6b31ea92a8; no voters:
I20260928 08:35:30.338121 3246 raft_consensus.cc:2781] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:35:30.338605 3246 raft_consensus.cc:686] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8 [term 1 LEADER]: Becoming Leader. State: Replica: ea562b70dd224e9aa7296a6b31ea92a8, State: Running, Role: LEADER
I20260928 08:35:30.339110 3246 consensus_queue.cc:237] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e34638302c4682bf6f0c5b04bd2f48" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33173 } } peers { permanent_uuid: "11d2d3cd11a54b2ea5ab43d98370a38b" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38299 } } peers { permanent_uuid: "ea562b70dd224e9aa7296a6b31ea92a8" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 40175 } }
I20260928 08:35:30.343712 2760 catalog_manager.cc:5495] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea562b70dd224e9aa7296a6b31ea92a8 (127.15.134.194). New cstate: current_term: 1 leader_uuid: "ea562b70dd224e9aa7296a6b31ea92a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e34638302c4682bf6f0c5b04bd2f48" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11d2d3cd11a54b2ea5ab43d98370a38b" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea562b70dd224e9aa7296a6b31ea92a8" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 40175 } health_report { overall_health: HEALTHY } } }
I20260928 08:35:30.360411 3236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:30.355194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47734 (local address 127.15.134.194:40175)
0928 08:35:30.355385 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:30.355390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:30.355563 (+ 173us) server_negotiation.cc:407] Connection header received
0928 08:35:30.355738 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:30.355742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:30.355821 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:30.355973 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:35:30.356271 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.357074 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.358497 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.358891 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.358993 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:30.359398 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:35:30.359894 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:35:30.360029 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:30.360137 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 08:35:30.360202 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 08:35:30.368981 2906 raft_consensus.cc:1261] T 3973adfac4244105b6942b03d9399da7 P 70e34638302c4682bf6f0c5b04bd2f48 [term 1 FOLLOWER]: Refusing update from remote peer ea562b70dd224e9aa7296a6b31ea92a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:35:30.368981 3174 raft_consensus.cc:1261] T 3973adfac4244105b6942b03d9399da7 P 11d2d3cd11a54b2ea5ab43d98370a38b [term 1 FOLLOWER]: Refusing update from remote peer ea562b70dd224e9aa7296a6b31ea92a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:35:30.370165 3249 consensus_queue.cc:1035] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11d2d3cd11a54b2ea5ab43d98370a38b" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 38299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:35:30.370560 3246 consensus_queue.cc:1035] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70e34638302c4682bf6f0c5b04bd2f48" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 33173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:35:30.381603 3252 mvcc.cc:205] Tried to move back new op lower bound from 7334234236380200960 to 7334234236273082368. Current Snapshot: MvccSnapshot[applied={T|T < 7334234236380200960}]
I20260928 08:35:30.386389 3254 mvcc.cc:205] Tried to move back new op lower bound from 7334234236380200960 to 7334234236273082368. Current Snapshot: MvccSnapshot[applied={T|T < 7334234236380200960}]
I20260928 08:35:30.399482 2760 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:35:30.399818 2760 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:35:30.403703 2760 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 691984ee93e5487bba2d807a186d66d9: Sending DeleteTablet for 3 replicas of tablet 3973adfac4244105b6942b03d9399da7
I20260928 08:35:30.405061 3154 tablet_service.cc:1514] Processing DeleteTablet for tablet 3973adfac4244105b6942b03d9399da7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:35:30 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:42274
I20260928 08:35:30.405200 3020 tablet_service.cc:1514] Processing DeleteTablet for tablet 3973adfac4244105b6942b03d9399da7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:35:30 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:47720
I20260928 08:35:30.405203 2885 tablet_service.cc:1514] Processing DeleteTablet for tablet 3973adfac4244105b6942b03d9399da7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:35:30 UTC) from {username='kudu', principal='kudu/127.15.134.254@KRBTEST.COM'} at 127.0.0.1:43462
I20260928 08:35:30.407998 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 2822
I20260928 08:35:30.418560 3264 tablet_replica.cc:330] T 3973adfac4244105b6942b03d9399da7 P 70e34638302c4682bf6f0c5b04bd2f48: stopping tablet replica
I20260928 08:35:30.419075 3264 raft_consensus.cc:2229] T 3973adfac4244105b6942b03d9399da7 P 70e34638302c4682bf6f0c5b04bd2f48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:35:30.419621 3264 raft_consensus.cc:2258] T 3973adfac4244105b6942b03d9399da7 P 70e34638302c4682bf6f0c5b04bd2f48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:35:30.426131 3264 ts_tablet_manager.cc:1905] T 3973adfac4244105b6942b03d9399da7 P 70e34638302c4682bf6f0c5b04bd2f48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:35:30.436203 3264 ts_tablet_manager.cc:1918] T 3973adfac4244105b6942b03d9399da7 P 70e34638302c4682bf6f0c5b04bd2f48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:35:30.440449 3262 tablet_replica.cc:330] T 3973adfac4244105b6942b03d9399da7 P 11d2d3cd11a54b2ea5ab43d98370a38b: stopping tablet replica
I20260928 08:35:30.440774 3263 tablet_replica.cc:330] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8: stopping tablet replica
I20260928 08:35:30.441002 3262 raft_consensus.cc:2229] T 3973adfac4244105b6942b03d9399da7 P 11d2d3cd11a54b2ea5ab43d98370a38b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:35:30.441197 3263 raft_consensus.cc:2229] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:35:30.441566 3262 raft_consensus.cc:2258] T 3973adfac4244105b6942b03d9399da7 P 11d2d3cd11a54b2ea5ab43d98370a38b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:35:30.441919 3263 raft_consensus.cc:2258] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:35:30.443689 3264 log.cc:1196] T 3973adfac4244105b6942b03d9399da7 P 70e34638302c4682bf6f0c5b04bd2f48: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-0/wal/wals/3973adfac4244105b6942b03d9399da7
I20260928 08:35:30.444533 3262 ts_tablet_manager.cc:1905] T 3973adfac4244105b6942b03d9399da7 P 11d2d3cd11a54b2ea5ab43d98370a38b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:35:30.444540 3263 ts_tablet_manager.cc:1905] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 08:35:30 dist-test-slave-qp2n krb5kdc[2716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.193@KRBTEST.COM
I20260928 08:35:30.447948 3264 ts_tablet_manager.cc:1939] T 3973adfac4244105b6942b03d9399da7 P 70e34638302c4682bf6f0c5b04bd2f48: Deleting consensus metadata
I20260928 08:35:30.451256 2746 catalog_manager.cc:4841] TS 70e34638302c4682bf6f0c5b04bd2f48 (127.15.134.193:33173): tablet 3973adfac4244105b6942b03d9399da7 (table test-table [id=3208e460374a4e61a65cc970c6a3a791]) successfully deleted
I20260928 08:35:30.452165 3263 ts_tablet_manager.cc:1918] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:35:30.452419 3263 log.cc:1196] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/3973adfac4244105b6942b03d9399da7
I20260928 08:35:30.453075 3263 ts_tablet_manager.cc:1939] T 3973adfac4244105b6942b03d9399da7 P ea562b70dd224e9aa7296a6b31ea92a8: Deleting consensus metadata
I20260928 08:35:30.454938 2745 catalog_manager.cc:4841] TS ea562b70dd224e9aa7296a6b31ea92a8 (127.15.134.194:40175): tablet 3973adfac4244105b6942b03d9399da7 (table test-table [id=3208e460374a4e61a65cc970c6a3a791]) successfully deleted
I20260928 08:35:30.455967 3262 ts_tablet_manager.cc:1918] T 3973adfac4244105b6942b03d9399da7 P 11d2d3cd11a54b2ea5ab43d98370a38b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:35:30.456230 3262 log.cc:1196] T 3973adfac4244105b6942b03d9399da7 P 11d2d3cd11a54b2ea5ab43d98370a38b: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/3973adfac4244105b6942b03d9399da7
I20260928 08:35:30.456815 3262 ts_tablet_manager.cc:1939] T 3973adfac4244105b6942b03d9399da7 P 11d2d3cd11a54b2ea5ab43d98370a38b: Deleting consensus metadata
I20260928 08:35:30.458381 2748 catalog_manager.cc:4841] TS 11d2d3cd11a54b2ea5ab43d98370a38b (127.15.134.195:38299): tablet 3973adfac4244105b6942b03d9399da7 (table test-table [id=3208e460374a4e61a65cc970c6a3a791]) successfully deleted
I20260928 08:35:30.459251 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:30.409757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43464 (local address 127.15.134.193:33173)
0928 08:35:30.409995 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:30.410005 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:30.432730 (+ 22725us) server_negotiation.cc:407] Connection header received
0928 08:35:30.433117 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:30.433126 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:30.433261 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:30.433353 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:30.435762 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.437419 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.441786 (+ 4367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.442402 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.442527 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:30.451763 (+ 9236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:30.451778 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:30.451794 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:30.451836 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:30.455632 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:30.456564 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:30.456576 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:30.456581 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:30.456776 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:30.457268 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:30.457273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:30.457277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:30.457735 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:30.458016 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:30.458642 (+ 626us) server_negotiation.cc:299] Negotiation successful
0928 08:35:30.458879 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 08:35:30.460708 2946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:30.596379 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 2822
I20260928 08:35:30.615461 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 2957
Sep 28 08:35:30 dist-test-slave-qp2n krb5kdc[2716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.194@KRBTEST.COM
I20260928 08:35:30.633702 3236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:30.616345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47742 (local address 127.15.134.194:40175)
0928 08:35:30.616569 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:30.616575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:30.616597 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:35:30.616717 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:30.616723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:30.616812 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:30.616921 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:30.618157 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.619399 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.620740 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.621432 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.621577 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:30.626408 (+ 4831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:30.626434 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:30.626469 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:30.626526 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:30.630520 (+ 3994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:30.631433 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:30.631440 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:30.631442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:30.631583 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:30.632078 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:30.632085 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:30.632089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:30.632509 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:30.632837 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:30.633139 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 08:35:30.633395 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 08:35:30.634701 3080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:30.750322 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 2957
I20260928 08:35:30.771353 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 3091
Sep 28 08:35:30 dist-test-slave-qp2n krb5kdc[2716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.195@KRBTEST.COM
I20260928 08:35:30.788906 3237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:30.772489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42286 (local address 127.15.134.195:38299)
0928 08:35:30.772722 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:30.772731 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:30.772756 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:35:30.773009 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:30.773013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:30.773087 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:30.773197 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:30.774580 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.775300 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.776588 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:30.777059 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:30.777190 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:30.782531 (+ 5341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:30.782550 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:30.782567 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:30.782614 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:30.785708 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:30.786457 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:30.786464 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:30.786467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:30.786624 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:30.787057 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:30.787064 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:30.787068 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:30.787487 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:30.787846 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:30.788248 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 08:35:30.788500 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 08:35:30.789639 3214 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:30.880365 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 3091
I20260928 08:35:30.898795 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 2729
I20260928 08:35:30.900115 2791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:31.013963 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 2729
2026-09-28T08:35:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4589 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T08:35:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:35:31Z Disabled control of system clock
I20260928 08:35:31.097160 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:36443
--webserver_interface=127.15.134.254
--webserver_port=0
--builtin_ntp_servers=127.15.134.212:33189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:36443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:35:31.441223 3273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:31.441519 3273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:31.441649 3273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:31.449123 3273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:35:31.449232 3273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:31.449312 3273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:31.449426 3273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:35:31.449504 3273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:35:31.458994 3273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:33189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:36443
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:36443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:31.460400 3273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:31.462426 3273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:31.474908 3278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:31.475028 3281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:31.474896 3279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:31.476063 3273 server_base.cc:1034] running on GCE node
I20260928 08:35:31.476976 3273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:31.478588 3273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:31.479885 3273 hybrid_clock.cc:648] HybridClock initialized: now 1790584531479806 us; error 47 us; skew 500 ppm
I20260928 08:35:31.480345 3273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:35:31.483028 3273 webserver.cc:460] Webserver started at http://127.15.134.254:35141/ using document root <none> and password file <none>
I20260928 08:35:31.483817 3273 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:31.483963 3273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:31.484298 3273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:31.488030 3273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "dc9ceb63cbf545829000ff02f14441c7"
format_stamp: "Formatted at 2026-09-28 08:35:31 on dist-test-slave-qp2n"
server_key: "5842885ad9078479147db668cdf56c16"
server_key_iv: "e89040a51b23eda81c4fe9a0c54f8a56"
server_key_version: "encryptionkey@0"
I20260928 08:35:31.489091 3273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "dc9ceb63cbf545829000ff02f14441c7"
format_stamp: "Formatted at 2026-09-28 08:35:31 on dist-test-slave-qp2n"
server_key: "5842885ad9078479147db668cdf56c16"
server_key_iv: "e89040a51b23eda81c4fe9a0c54f8a56"
server_key_version: "encryptionkey@0"
I20260928 08:35:31.495283 3273 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260928 08:35:31.499235 3287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:31.500763 3273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 08:35:31.500973 3273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "dc9ceb63cbf545829000ff02f14441c7"
format_stamp: "Formatted at 2026-09-28 08:35:31 on dist-test-slave-qp2n"
server_key: "5842885ad9078479147db668cdf56c16"
server_key_iv: "e89040a51b23eda81c4fe9a0c54f8a56"
server_key_version: "encryptionkey@0"
I20260928 08:35:31.501204 3273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:31.518070 3273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:31.518971 3273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:31.519263 3273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:31.538928 3273 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:36443
I20260928 08:35:31.538939 3338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:36443 every 8 connection(s)
I20260928 08:35:31.540251 3273 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:35:31.542868 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 3273
I20260928 08:35:31.543046 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:35:31.543510 15899 external_mini_cluster.cc:1428] Setting key 7268a270f32dae533e579c42e7df463c
I20260928 08:35:31.544138 3339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:31.559429 3342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:31.545446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57864 (local address 127.15.134.254:36443)
0928 08:35:31.546520 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:31.546533 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:31.546581 (+ 48us) server_negotiation.cc:407] Connection header received
0928 08:35:31.548988 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:31.549018 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:31.549645 (+ 627us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:31.550087 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:31.550654 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:31.552666 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:31.554181 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:31.555250 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:31.555468 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:31.556562 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:31.556572 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:31.556591 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:31.557391 (+ 800us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:31.557498 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:31.557524 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:31.557634 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:31.557834 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 08:35:31.558157 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":372,"threads_started":1}
I20260928 08:35:31.559615 3339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc9ceb63cbf545829000ff02f14441c7: Bootstrap starting.
I20260928 08:35:31.562876 3339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc9ceb63cbf545829000ff02f14441c7: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:31.563863 3339 log.cc:824] T 00000000000000000000000000000000 P dc9ceb63cbf545829000ff02f14441c7: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:31.566638 3339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc9ceb63cbf545829000ff02f14441c7: No bootstrap required, opened a new log
I20260928 08:35:31.574829 3339 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc9ceb63cbf545829000ff02f14441c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9ceb63cbf545829000ff02f14441c7" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 36443 } }
I20260928 08:35:31.575186 3339 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc9ceb63cbf545829000ff02f14441c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:31.575263 3339 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc9ceb63cbf545829000ff02f14441c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9ceb63cbf545829000ff02f14441c7, State: Initialized, Role: FOLLOWER
I20260928 08:35:31.575742 3339 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc9ceb63cbf545829000ff02f14441c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9ceb63cbf545829000ff02f14441c7" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 36443 } }
I20260928 08:35:31.575946 3339 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc9ceb63cbf545829000ff02f14441c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:35:31.576040 3339 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc9ceb63cbf545829000ff02f14441c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:35:31.576161 3339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc9ceb63cbf545829000ff02f14441c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:31.578617 3339 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc9ceb63cbf545829000ff02f14441c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9ceb63cbf545829000ff02f14441c7" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 36443 } }
I20260928 08:35:31.579102 3339 leader_election.cc:302] T 00000000000000000000000000000000 P dc9ceb63cbf545829000ff02f14441c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc9ceb63cbf545829000ff02f14441c7; no voters:
I20260928 08:35:31.579700 3339 leader_election.cc:288] T 00000000000000000000000000000000 P dc9ceb63cbf545829000ff02f14441c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:35:31.580071 3344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc9ceb63cbf545829000ff02f14441c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:35:31.580919 3344 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc9ceb63cbf545829000ff02f14441c7 [term 1 LEADER]: Becoming Leader. State: Replica: dc9ceb63cbf545829000ff02f14441c7, State: Running, Role: LEADER
I20260928 08:35:31.581477 3344 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc9ceb63cbf545829000ff02f14441c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9ceb63cbf545829000ff02f14441c7" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 36443 } }
I20260928 08:35:31.582050 3339 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc9ceb63cbf545829000ff02f14441c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:35:31.584414 3345 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc9ceb63cbf545829000ff02f14441c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc9ceb63cbf545829000ff02f14441c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9ceb63cbf545829000ff02f14441c7" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 36443 } } }
I20260928 08:35:31.584894 3345 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc9ceb63cbf545829000ff02f14441c7 [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:31.585386 3346 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc9ceb63cbf545829000ff02f14441c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc9ceb63cbf545829000ff02f14441c7. Latest consensus state: current_term: 1 leader_uuid: "dc9ceb63cbf545829000ff02f14441c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc9ceb63cbf545829000ff02f14441c7" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 36443 } } }
I20260928 08:35:31.585803 3346 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc9ceb63cbf545829000ff02f14441c7 [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:31.587098 3351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:35:31.593638 3351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:35:31.602710 3351 catalog_manager.cc:1347] Generated new cluster ID: 2796697b160142888e2a5b52a796cb25
I20260928 08:35:31.602850 3351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:35:31.616822 3351 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:35:31.618189 3351 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:35:31.627992 3351 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc9ceb63cbf545829000ff02f14441c7: Generated new TSK 0
I20260928 08:35:31.629139 3351 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:35:31.637779 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.193:0
--local_ip_for_outbound_sockets=127.15.134.193
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:36443
--builtin_ntp_servers=127.15.134.212:33189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:35:32.008349 3363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:32.008809 3363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:32.008993 3363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:32.017041 3363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:32.017179 3363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:32.017359 3363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.193
I20260928 08:35:32.026927 3363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:33189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.15.134.193
--webserver_port=0
--tserver_master_addrs=127.15.134.254:36443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.193
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:32.028393 3363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:32.030319 3363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:32.043937 3368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:32.044029 3369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:32.044546 3371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:32.050272 3363 server_base.cc:1034] running on GCE node
I20260928 08:35:32.051043 3363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:32.052104 3363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:32.053363 3363 hybrid_clock.cc:648] HybridClock initialized: now 1790584532053304 us; error 54 us; skew 500 ppm
I20260928 08:35:32.053741 3363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:35:32.056442 3363 webserver.cc:460] Webserver started at http://127.15.134.193:35063/ using document root <none> and password file <none>
I20260928 08:35:32.057464 3363 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:32.057610 3363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:32.058123 3363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:32.061645 3363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-0/data/instance:
uuid: "84ba6c788b2f4d0093e50c745a4f3e19"
format_stamp: "Formatted at 2026-09-28 08:35:32 on dist-test-slave-qp2n"
server_key: "d579278a173982263cfcb15bf9cdfa6c"
server_key_iv: "dfbabb204b660e0e7931b67fae1fafb9"
server_key_version: "encryptionkey@0"
I20260928 08:35:32.062559 3363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance:
uuid: "84ba6c788b2f4d0093e50c745a4f3e19"
format_stamp: "Formatted at 2026-09-28 08:35:32 on dist-test-slave-qp2n"
server_key: "d579278a173982263cfcb15bf9cdfa6c"
server_key_iv: "dfbabb204b660e0e7931b67fae1fafb9"
server_key_version: "encryptionkey@0"
I20260928 08:35:32.068928 3363 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260928 08:35:32.073268 3377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:32.074966 3363 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 08:35:32.075174 3363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-0/wal
uuid: "84ba6c788b2f4d0093e50c745a4f3e19"
format_stamp: "Formatted at 2026-09-28 08:35:32 on dist-test-slave-qp2n"
server_key: "d579278a173982263cfcb15bf9cdfa6c"
server_key_iv: "dfbabb204b660e0e7931b67fae1fafb9"
server_key_version: "encryptionkey@0"
I20260928 08:35:32.075455 3363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:32.094779 3363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:32.095681 3363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:32.095997 3363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:32.097080 3363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:35:32.098551 3363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:35:32.098680 3363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:32.098807 3363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:35:32.098886 3363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:32.133078 3363 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.193:35979
I20260928 08:35:32.133169 3489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.193:35979 every 8 connection(s)
I20260928 08:35:32.134531 3363 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-0/data/info.pb
I20260928 08:35:32.139562 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 3363
I20260928 08:35:32.139736 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-0/wal/instance
I20260928 08:35:32.140234 15899 external_mini_cluster.cc:1428] Setting key ff530da03d13a80c16d69b71d3e7d046
I20260928 08:35:32.143472 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.194:0
--local_ip_for_outbound_sockets=127.15.134.194
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:36443
--builtin_ntp_servers=127.15.134.212:33189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 08:35:32.148599 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:32.137240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.193:45243 (local address 127.15.134.254:36443)
0928 08:35:32.137660 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:32.137665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:32.138941 (+ 1276us) server_negotiation.cc:407] Connection header received
0928 08:35:32.141045 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:32.141052 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:32.141140 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:32.141266 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:32.142690 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:32.143780 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:32.145444 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:32.146145 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:32.146289 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:32.147605 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:32.147614 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:32.147622 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:32.147648 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:32.147699 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:32.147719 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:32.147814 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:32.148028 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 08:35:32.148136 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":202,"threads_started":1}
I20260928 08:35:32.149457 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:32.137866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:36443 (local address 127.15.134.193:45243)
0928 08:35:32.138751 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:32.138814 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:32.140703 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:32.141572 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:32.141583 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:32.141698 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:35:32.142489 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:32.142510 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:32.143943 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:32.143947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:32.145290 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:32.145303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:32.146363 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:32.146368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:32.146564 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:32.147268 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:35:32.147270 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:35:32.147309 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:35:32.147408 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:35:32.147816 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:35:32.147824 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:35:32.147830 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 08:35:32.147984 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 08:35:32.148280 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":265,"threads_started":1}
I20260928 08:35:32.151052 3490 heartbeater.cc:347] Connected to a master server at 127.15.134.254:36443
I20260928 08:35:32.151394 3490 heartbeater.cc:464] Registering TS with master...
I20260928 08:35:32.152267 3490 heartbeater.cc:511] Master 127.15.134.254:36443 requested a full tablet report, sending...
I20260928 08:35:32.154662 3304 ts_manager.cc:194] Registered new tserver with Master: 84ba6c788b2f4d0093e50c745a4f3e19 (127.15.134.193:35979)
I20260928 08:35:32.156906 3304 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.134.193:45243
W20260928 08:35:32.486281 3494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:32.486589 3494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:32.486677 3494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:32.494520 3494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:32.494627 3494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:32.494733 3494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.194
I20260928 08:35:32.503933 3494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:33189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.15.134.194
--webserver_port=0
--tserver_master_addrs=127.15.134.254:36443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.194
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:32.505251 3494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:32.507099 3494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:32.520529 3499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:32.520454 3502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:32.520494 3500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:32.521291 3494 server_base.cc:1034] running on GCE node
I20260928 08:35:32.521961 3494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:32.522974 3494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:32.524253 3494 hybrid_clock.cc:648] HybridClock initialized: now 1790584532524186 us; error 47 us; skew 500 ppm
I20260928 08:35:32.524668 3494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:35:32.527089 3494 webserver.cc:460] Webserver started at http://127.15.134.194:45911/ using document root <none> and password file <none>
I20260928 08:35:32.527817 3494 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:32.527966 3494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:32.528309 3494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:32.531404 3494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-1/data/instance:
uuid: "43203538ba614070bdbc04cf8a0585c3"
format_stamp: "Formatted at 2026-09-28 08:35:32 on dist-test-slave-qp2n"
server_key: "b797623b53dc74454fd81bfb3cc31f4a"
server_key_iv: "e55382dd1681635f254264c0f2593f4d"
server_key_version: "encryptionkey@0"
I20260928 08:35:32.532238 3494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance:
uuid: "43203538ba614070bdbc04cf8a0585c3"
format_stamp: "Formatted at 2026-09-28 08:35:32 on dist-test-slave-qp2n"
server_key: "b797623b53dc74454fd81bfb3cc31f4a"
server_key_iv: "e55382dd1681635f254264c0f2593f4d"
server_key_version: "encryptionkey@0"
I20260928 08:35:32.538158 3494 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.007s
I20260928 08:35:32.542666 3508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:32.544309 3494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:35:32.544468 3494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-1/wal
uuid: "43203538ba614070bdbc04cf8a0585c3"
format_stamp: "Formatted at 2026-09-28 08:35:32 on dist-test-slave-qp2n"
server_key: "b797623b53dc74454fd81bfb3cc31f4a"
server_key_iv: "e55382dd1681635f254264c0f2593f4d"
server_key_version: "encryptionkey@0"
I20260928 08:35:32.544739 3494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:32.572741 3494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:32.573688 3494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:32.574024 3494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:32.575037 3494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:35:32.576687 3494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:35:32.576802 3494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:32.576941 3494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:35:32.577029 3494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:32.614290 3494 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.194:34421
I20260928 08:35:32.614400 3620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.194:34421 every 8 connection(s)
I20260928 08:35:32.615760 3494 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-1/data/info.pb
I20260928 08:35:32.623132 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 3494
I20260928 08:35:32.623407 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-1/wal/instance
I20260928 08:35:32.623914 15899 external_mini_cluster.cc:1428] Setting key 9dbd481179f65e6f65f231d116e93560
I20260928 08:35:32.626876 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.134.195:0
--local_ip_for_outbound_sockets=127.15.134.195
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:36443
--builtin_ntp_servers=127.15.134.212:33189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 08:35:32.630023 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:32.618715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.194:36051 (local address 127.15.134.254:36443)
0928 08:35:32.618972 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:32.618977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:32.620453 (+ 1476us) server_negotiation.cc:407] Connection header received
0928 08:35:32.622545 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:32.622553 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:32.622656 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:32.622791 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:32.624129 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:32.625131 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:32.626853 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:32.627372 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:32.627533 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:32.629107 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:32.629113 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:32.629116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:32.629136 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:32.629169 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:32.629182 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:32.629291 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:32.629536 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 08:35:32.629649 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260928 08:35:32.631161 3623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:32.619236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:36443 (local address 127.15.134.194:36051)
0928 08:35:32.620270 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:32.620321 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:32.622258 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:32.623036 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:32.623044 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:32.623137 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:35:32.623902 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:32.623920 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:32.625295 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:32.625301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:32.626691 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:32.626704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:32.627619 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:32.627627 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:32.627892 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:32.628748 (+ 856us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:35:32.628753 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:35:32.628797 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:35:32.628888 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:35:32.629293 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:35:32.629308 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:35:32.629319 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 08:35:32.629523 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 08:35:32.629959 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":312,"threads_started":1}
I20260928 08:35:32.632900 3621 heartbeater.cc:347] Connected to a master server at 127.15.134.254:36443
I20260928 08:35:32.633219 3621 heartbeater.cc:464] Registering TS with master...
I20260928 08:35:32.634387 3621 heartbeater.cc:511] Master 127.15.134.254:36443 requested a full tablet report, sending...
I20260928 08:35:32.636247 3304 ts_manager.cc:194] Registered new tserver with Master: 43203538ba614070bdbc04cf8a0585c3 (127.15.134.194:34421)
I20260928 08:35:32.637375 3304 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.134.194:36051
W20260928 08:35:32.965417 3624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:32.965710 3624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:32.965807 3624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:32.972958 3624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:32.973064 3624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:32.973237 3624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.134.195
I20260928 08:35:32.982916 3624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:33189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.134.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.15.134.195
--webserver_port=0
--tserver_master_addrs=127.15.134.254:36443
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.134.195
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:32.984458 3624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:32.986346 3624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:33.000155 3632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:33.000138 3629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:33.000264 3630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:33.006579 3624 server_base.cc:1034] running on GCE node
I20260928 08:35:33.007324 3624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:33.008394 3624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:33.009653 3624 hybrid_clock.cc:648] HybridClock initialized: now 1790584533009599 us; error 69 us; skew 500 ppm
I20260928 08:35:33.010012 3624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:35:33.012523 3624 webserver.cc:460] Webserver started at http://127.15.134.195:43093/ using document root <none> and password file <none>
I20260928 08:35:33.013314 3624 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:33.013453 3624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:33.013808 3624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:33.016812 3624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-2/data/instance:
uuid: "1532098bb09448c5a561b53a029108c3"
format_stamp: "Formatted at 2026-09-28 08:35:33 on dist-test-slave-qp2n"
server_key: "e012f8a75e9e7f8ec2fef96c25a19f36"
server_key_iv: "b7d2caffcb68ee4877a681128c6a3b4f"
server_key_version: "encryptionkey@0"
I20260928 08:35:33.017639 3624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance:
uuid: "1532098bb09448c5a561b53a029108c3"
format_stamp: "Formatted at 2026-09-28 08:35:33 on dist-test-slave-qp2n"
server_key: "e012f8a75e9e7f8ec2fef96c25a19f36"
server_key_iv: "b7d2caffcb68ee4877a681128c6a3b4f"
server_key_version: "encryptionkey@0"
I20260928 08:35:33.022711 3624 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:35:33.026257 3638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:33.027577 3624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:35:33.027751 3624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-2/wal
uuid: "1532098bb09448c5a561b53a029108c3"
format_stamp: "Formatted at 2026-09-28 08:35:33 on dist-test-slave-qp2n"
server_key: "e012f8a75e9e7f8ec2fef96c25a19f36"
server_key_iv: "b7d2caffcb68ee4877a681128c6a3b4f"
server_key_version: "encryptionkey@0"
I20260928 08:35:33.027969 3624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:33.041115 3624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:33.041952 3624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:33.042285 3624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:33.043257 3624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:35:33.044770 3624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:35:33.044889 3624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:33.045024 3624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:35:33.045116 3624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:33.079115 3624 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.195:39389
I20260928 08:35:33.079203 3750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.195:39389 every 8 connection(s)
I20260928 08:35:33.080507 3624 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-2/data/info.pb
I20260928 08:35:33.085533 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 3624
I20260928 08:35:33.085736 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-2/wal/instance
I20260928 08:35:33.086165 15899 external_mini_cluster.cc:1428] Setting key ca38d28d74b455a4e8d4d3460f8bb51c
I20260928 08:35:33.094621 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:33.083346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:41283 (local address 127.15.134.254:36443)
0928 08:35:33.083560 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:33.083566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:33.085207 (+ 1641us) server_negotiation.cc:407] Connection header received
0928 08:35:33.087419 (+ 2212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:33.087425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:33.087522 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:33.087606 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:33.089012 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.089955 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.091604 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.092032 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.092131 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:33.093663 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:33.093668 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:33.093671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:33.093690 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:33.093722 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:33.093734 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:33.093824 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:33.094260 (+ 436us) server_negotiation.cc:299] Negotiation successful
0928 08:35:33.094360 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 08:35:33.095757 3753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:33.083919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:36443 (local address 127.15.134.195:41283)
0928 08:35:33.085045 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:33.085084 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:33.087114 (+ 2030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:33.087903 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:33.087911 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:33.088049 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:35:33.088801 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:33.088821 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.090104 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.090108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:33.091441 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:33.091453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.092236 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.092242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:33.092467 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:33.093314 (+ 847us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:35:33.093317 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:35:33.093365 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:35:33.093451 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:35:33.094024 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:35:33.094033 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:35:33.094039 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 08:35:33.094217 (+ 178us) client_negotiation.cc:240] Negotiation successful
0928 08:35:33.094546 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":315,"threads_started":1}
I20260928 08:35:33.097330 3751 heartbeater.cc:347] Connected to a master server at 127.15.134.254:36443
I20260928 08:35:33.097683 3751 heartbeater.cc:464] Registering TS with master...
I20260928 08:35:33.098549 3751 heartbeater.cc:511] Master 127.15.134.254:36443 requested a full tablet report, sending...
I20260928 08:35:33.100224 3304 ts_manager.cc:194] Registered new tserver with Master: 1532098bb09448c5a561b53a029108c3 (127.15.134.195:39389)
I20260928 08:35:33.101322 3304 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.134.195:41283
I20260928 08:35:33.102604 15899 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:35:33.113680 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:33.107476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57876 (local address 127.15.134.254:36443)
0928 08:35:33.107685 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:33.107690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:33.108162 (+ 472us) server_negotiation.cc:407] Connection header received
0928 08:35:33.108329 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:33.108333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:33.108403 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:33.108485 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:33.109033 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.109774 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.111087 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.111503 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.111601 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:33.112797 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:33.112802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:33.112806 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:33.112826 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:33.112862 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:33.112875 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:33.112960 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:33.113157 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 08:35:33.113275 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 08:35:33.118875 3303 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:35:33.126317 3303 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:35:33.160147 3490 heartbeater.cc:503] Master 127.15.134.254:36443 was elected leader, sending a full tablet report...
I20260928 08:35:33.163436 3763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:33.140065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:34421 (local address 127.0.0.1:58260)
0928 08:35:33.152133 (+ 12068us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:33.152179 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:33.152386 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:33.153419 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:33.153430 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:33.153498 (+ 68us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:33.154090 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:33.154108 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.156906 (+ 2798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.156919 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:33.159991 (+ 3072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:33.160007 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.162431 (+ 2424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.162438 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:33.162658 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:33.162713 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 08:35:33.162863 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 08:35:33.163066 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11811,"thread_start_us":11651,"threads_started":1}
I20260928 08:35:33.163868 3765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:33.140310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58260 (local address 127.15.134.194:34421)
0928 08:35:33.151640 (+ 11330us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:33.151654 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:33.152709 (+ 1055us) server_negotiation.cc:407] Connection header received
0928 08:35:33.152827 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:33.152839 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:33.153124 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:33.154355 (+ 1231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:33.154644 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.156686 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.160294 (+ 3608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.162193 (+ 1899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.162441 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:33.162743 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:33.163314 (+ 571us) server_negotiation.cc:299] Negotiation successful
0928 08:35:33.163511 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11078,"thread_start_us":10908,"threads_started":1}
I20260928 08:35:33.166699 3764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:33.140141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:35979 (local address 127.0.0.1:46410)
0928 08:35:33.157163 (+ 17022us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:33.157179 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:33.157346 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:33.158429 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:33.158433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:33.158463 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:33.158833 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:33.158845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.160756 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.160761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:33.163176 (+ 2415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:33.163190 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.165917 (+ 2727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.165923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:33.166100 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:33.166150 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 08:35:33.166238 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 08:35:33.166376 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16867,"thread_start_us":16701,"threads_started":1}
I20260928 08:35:33.166806 3766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:33.140692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46410 (local address 127.15.134.193:35979)
0928 08:35:33.153687 (+ 12995us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:33.153699 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:33.157516 (+ 3817us) server_negotiation.cc:407] Connection header received
0928 08:35:33.157646 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:33.157655 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:33.157951 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:33.158168 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:33.159020 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.160576 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.164385 (+ 3809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.165698 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.165923 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:33.166128 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:33.166299 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 08:35:33.166468 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12734,"thread_start_us":13001,"threads_started":1}
I20260928 08:35:33.168354 3556 tablet_service.cc:1467] Processing CreateTablet for tablet e83d606dad3c4429a76dac1d522194b9 (DEFAULT_TABLE table=test-table [id=5c9db0de29b547bb946ad85ffa0fa183]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:35:33.169759 3556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e83d606dad3c4429a76dac1d522194b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:33.170320 3425 tablet_service.cc:1467] Processing CreateTablet for tablet e83d606dad3c4429a76dac1d522194b9 (DEFAULT_TABLE table=test-table [id=5c9db0de29b547bb946ad85ffa0fa183]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:35:33.171195 3768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:33.159206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.195:39389 (local address 127.0.0.1:54734)
0928 08:35:33.159836 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:33.159855 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:33.159961 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:33.164255 (+ 4294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:33.164263 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:33.164294 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:33.164605 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:33.164644 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.166230 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.166236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:33.168490 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:33.168504 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.170316 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.170322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:33.170485 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:33.170516 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 08:35:33.170603 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 08:35:33.170757 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"spinlock_wait_cycles":6400,"thread_start_us":459,"threads_started":1}
I20260928 08:35:33.171231 3767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:33.153058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54734 (local address 127.15.134.195:39389)
0928 08:35:33.163314 (+ 10256us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:33.163325 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:33.163366 (+ 41us) server_negotiation.cc:407] Connection header received
0928 08:35:33.163473 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:33.163481 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:33.163749 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:33.163975 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:33.164821 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.166059 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.168730 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.170107 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.170314 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:33.170539 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:33.170668 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 08:35:33.170833 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10075,"thread_start_us":9911,"threads_started":1}
I20260928 08:35:33.171820 3425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e83d606dad3c4429a76dac1d522194b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:33.175320 3686 tablet_service.cc:1467] Processing CreateTablet for tablet e83d606dad3c4429a76dac1d522194b9 (DEFAULT_TABLE table=test-table [id=5c9db0de29b547bb946ad85ffa0fa183]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:35:33.176903 3686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e83d606dad3c4429a76dac1d522194b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:33.184087 3769 tablet_bootstrap.cc:492] T e83d606dad3c4429a76dac1d522194b9 P 84ba6c788b2f4d0093e50c745a4f3e19: Bootstrap starting.
I20260928 08:35:33.187464 3769 tablet_bootstrap.cc:654] T e83d606dad3c4429a76dac1d522194b9 P 84ba6c788b2f4d0093e50c745a4f3e19: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:33.188520 3769 log.cc:824] T e83d606dad3c4429a76dac1d522194b9 P 84ba6c788b2f4d0093e50c745a4f3e19: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:33.189721 3770 tablet_bootstrap.cc:492] T e83d606dad3c4429a76dac1d522194b9 P 43203538ba614070bdbc04cf8a0585c3: Bootstrap starting.
I20260928 08:35:33.192754 3772 tablet_bootstrap.cc:492] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3: Bootstrap starting.
I20260928 08:35:33.193817 3770 tablet_bootstrap.cc:654] T e83d606dad3c4429a76dac1d522194b9 P 43203538ba614070bdbc04cf8a0585c3: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:33.194701 3769 tablet_bootstrap.cc:492] T e83d606dad3c4429a76dac1d522194b9 P 84ba6c788b2f4d0093e50c745a4f3e19: No bootstrap required, opened a new log
I20260928 08:35:33.195013 3769 ts_tablet_manager.cc:1397] T e83d606dad3c4429a76dac1d522194b9 P 84ba6c788b2f4d0093e50c745a4f3e19: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260928 08:35:33.195113 3770 log.cc:824] T e83d606dad3c4429a76dac1d522194b9 P 43203538ba614070bdbc04cf8a0585c3: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:33.196161 3772 tablet_bootstrap.cc:654] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:33.197431 3772 log.cc:824] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:33.199146 3770 tablet_bootstrap.cc:492] T e83d606dad3c4429a76dac1d522194b9 P 43203538ba614070bdbc04cf8a0585c3: No bootstrap required, opened a new log
I20260928 08:35:33.199445 3770 ts_tablet_manager.cc:1397] T e83d606dad3c4429a76dac1d522194b9 P 43203538ba614070bdbc04cf8a0585c3: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 08:35:33.201579 3772 tablet_bootstrap.cc:492] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3: No bootstrap required, opened a new log
I20260928 08:35:33.201849 3772 ts_tablet_manager.cc:1397] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260928 08:35:33.206534 3769 raft_consensus.cc:348] T e83d606dad3c4429a76dac1d522194b9 P 84ba6c788b2f4d0093e50c745a4f3e19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43203538ba614070bdbc04cf8a0585c3" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34421 } } peers { permanent_uuid: "84ba6c788b2f4d0093e50c745a4f3e19" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 35979 } } peers { permanent_uuid: "1532098bb09448c5a561b53a029108c3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 39389 } }
I20260928 08:35:33.207028 3769 raft_consensus.cc:374] T e83d606dad3c4429a76dac1d522194b9 P 84ba6c788b2f4d0093e50c745a4f3e19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:33.207127 3769 raft_consensus.cc:729] T e83d606dad3c4429a76dac1d522194b9 P 84ba6c788b2f4d0093e50c745a4f3e19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84ba6c788b2f4d0093e50c745a4f3e19, State: Initialized, Role: FOLLOWER
I20260928 08:35:33.207790 3769 consensus_queue.cc:260] T e83d606dad3c4429a76dac1d522194b9 P 84ba6c788b2f4d0093e50c745a4f3e19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43203538ba614070bdbc04cf8a0585c3" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34421 } } peers { permanent_uuid: "84ba6c788b2f4d0093e50c745a4f3e19" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 35979 } } peers { permanent_uuid: "1532098bb09448c5a561b53a029108c3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 39389 } }
I20260928 08:35:33.209686 3769 ts_tablet_manager.cc:1428] T e83d606dad3c4429a76dac1d522194b9 P 84ba6c788b2f4d0093e50c745a4f3e19: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260928 08:35:33.210062 3770 raft_consensus.cc:348] T e83d606dad3c4429a76dac1d522194b9 P 43203538ba614070bdbc04cf8a0585c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43203538ba614070bdbc04cf8a0585c3" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34421 } } peers { permanent_uuid: "84ba6c788b2f4d0093e50c745a4f3e19" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 35979 } } peers { permanent_uuid: "1532098bb09448c5a561b53a029108c3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 39389 } }
I20260928 08:35:33.210502 3770 raft_consensus.cc:374] T e83d606dad3c4429a76dac1d522194b9 P 43203538ba614070bdbc04cf8a0585c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:33.210573 3770 raft_consensus.cc:729] T e83d606dad3c4429a76dac1d522194b9 P 43203538ba614070bdbc04cf8a0585c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43203538ba614070bdbc04cf8a0585c3, State: Initialized, Role: FOLLOWER
I20260928 08:35:33.210574 3772 raft_consensus.cc:348] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43203538ba614070bdbc04cf8a0585c3" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34421 } } peers { permanent_uuid: "84ba6c788b2f4d0093e50c745a4f3e19" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 35979 } } peers { permanent_uuid: "1532098bb09448c5a561b53a029108c3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 39389 } }
I20260928 08:35:33.211020 3772 raft_consensus.cc:374] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:33.211144 3772 raft_consensus.cc:729] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1532098bb09448c5a561b53a029108c3, State: Initialized, Role: FOLLOWER
I20260928 08:35:33.211158 3770 consensus_queue.cc:260] T e83d606dad3c4429a76dac1d522194b9 P 43203538ba614070bdbc04cf8a0585c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43203538ba614070bdbc04cf8a0585c3" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34421 } } peers { permanent_uuid: "84ba6c788b2f4d0093e50c745a4f3e19" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 35979 } } peers { permanent_uuid: "1532098bb09448c5a561b53a029108c3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 39389 } }
I20260928 08:35:33.211741 3772 consensus_queue.cc:260] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43203538ba614070bdbc04cf8a0585c3" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34421 } } peers { permanent_uuid: "84ba6c788b2f4d0093e50c745a4f3e19" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 35979 } } peers { permanent_uuid: "1532098bb09448c5a561b53a029108c3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 39389 } }
I20260928 08:35:33.212838 3751 heartbeater.cc:503] Master 127.15.134.254:36443 was elected leader, sending a full tablet report...
I20260928 08:35:33.213523 3772 ts_tablet_manager.cc:1428] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260928 08:35:33.214504 3621 heartbeater.cc:503] Master 127.15.134.254:36443 was elected leader, sending a full tablet report...
I20260928 08:35:33.215209 3770 ts_tablet_manager.cc:1428] T e83d606dad3c4429a76dac1d522194b9 P 43203538ba614070bdbc04cf8a0585c3: Time spent starting tablet: real 0.016s user 0.012s sys 0.005s
I20260928 08:35:33.313477 3777 raft_consensus.cc:482] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:35:33.313812 3777 raft_consensus.cc:504] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43203538ba614070bdbc04cf8a0585c3" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34421 } } peers { permanent_uuid: "84ba6c788b2f4d0093e50c745a4f3e19" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 35979 } } peers { permanent_uuid: "1532098bb09448c5a561b53a029108c3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 39389 } }
I20260928 08:35:33.315919 3777 leader_election.cc:288] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43203538ba614070bdbc04cf8a0585c3 (127.15.134.194:34421), 84ba6c788b2f4d0093e50c745a4f3e19 (127.15.134.193:35979)
I20260928 08:35:33.321874 3765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:33.316671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:52005 (local address 127.15.134.194:34421)
0928 08:35:33.317045 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:33.317054 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:33.317074 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:35:33.317156 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:33.317161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:33.317265 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:33.317357 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:33.317729 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.318399 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.320174 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.320983 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.321137 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:33.321209 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:33.321477 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 08:35:33.321595 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233}
I20260928 08:35:33.321926 3753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:33.316148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.194:34421 (local address 127.15.134.195:52005)
0928 08:35:33.316320 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:33.316341 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:33.316496 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:33.317368 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:33.317372 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:33.317394 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:33.317580 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:33.317588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.318541 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.318546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:33.320030 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:33.320041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.321128 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.321133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:33.321327 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:33.321350 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 08:35:33.321452 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 08:35:33.321590 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":3}
I20260928 08:35:33.322264 3766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:33.316599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.15.134.195:40665 (local address 127.15.134.193:35979)
0928 08:35:33.316866 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:33.316872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:33.317190 (+ 318us) server_negotiation.cc:407] Connection header received
0928 08:35:33.317383 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:33.317388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:33.317464 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:33.317593 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:35:33.318127 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.318829 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.320662 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.321409 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.321595 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:33.321659 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:33.321924 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 08:35:33.322044 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 08:35:33.322471 3778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:33.316292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.193:35979 (local address 127.15.134.195:40665)
0928 08:35:33.317071 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0928 08:35:33.317089 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:35:33.317212 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:35:33.317612 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:35:33.317617 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:35:33.317640 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:35:33.317975 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:33.317984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.319000 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.319004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:33.320447 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:35:33.320458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.321587 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.321593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:35:33.321761 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:33.321783 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 08:35:33.321888 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 08:35:33.322008 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":471,"threads_started":1}
I20260928 08:35:33.323551 3576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e83d606dad3c4429a76dac1d522194b9" candidate_uuid: "1532098bb09448c5a561b53a029108c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43203538ba614070bdbc04cf8a0585c3" is_pre_election: true
I20260928 08:35:33.323663 3445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e83d606dad3c4429a76dac1d522194b9" candidate_uuid: "1532098bb09448c5a561b53a029108c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84ba6c788b2f4d0093e50c745a4f3e19" is_pre_election: true
I20260928 08:35:33.324221 3576 raft_consensus.cc:2454] T e83d606dad3c4429a76dac1d522194b9 P 43203538ba614070bdbc04cf8a0585c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1532098bb09448c5a561b53a029108c3 in term 0.
I20260928 08:35:33.324368 3445 raft_consensus.cc:2454] T e83d606dad3c4429a76dac1d522194b9 P 84ba6c788b2f4d0093e50c745a4f3e19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1532098bb09448c5a561b53a029108c3 in term 0.
I20260928 08:35:33.325018 3642 leader_election.cc:302] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1532098bb09448c5a561b53a029108c3, 43203538ba614070bdbc04cf8a0585c3; no voters:
I20260928 08:35:33.325416 3777 raft_consensus.cc:2781] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:35:33.325564 3777 raft_consensus.cc:482] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:35:33.325685 3777 raft_consensus.cc:3037] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:33.327869 3777 raft_consensus.cc:504] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43203538ba614070bdbc04cf8a0585c3" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34421 } } peers { permanent_uuid: "84ba6c788b2f4d0093e50c745a4f3e19" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 35979 } } peers { permanent_uuid: "1532098bb09448c5a561b53a029108c3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 39389 } }
I20260928 08:35:33.328534 3777 leader_election.cc:288] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3 [CANDIDATE]: Term 1 election: Requested vote from peers 43203538ba614070bdbc04cf8a0585c3 (127.15.134.194:34421), 84ba6c788b2f4d0093e50c745a4f3e19 (127.15.134.193:35979)
I20260928 08:35:33.328934 3576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e83d606dad3c4429a76dac1d522194b9" candidate_uuid: "1532098bb09448c5a561b53a029108c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43203538ba614070bdbc04cf8a0585c3"
I20260928 08:35:33.329145 3576 raft_consensus.cc:3037] T e83d606dad3c4429a76dac1d522194b9 P 43203538ba614070bdbc04cf8a0585c3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:33.329124 3445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e83d606dad3c4429a76dac1d522194b9" candidate_uuid: "1532098bb09448c5a561b53a029108c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84ba6c788b2f4d0093e50c745a4f3e19"
I20260928 08:35:33.329358 3445 raft_consensus.cc:3037] T e83d606dad3c4429a76dac1d522194b9 P 84ba6c788b2f4d0093e50c745a4f3e19 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:33.331509 3576 raft_consensus.cc:2454] T e83d606dad3c4429a76dac1d522194b9 P 43203538ba614070bdbc04cf8a0585c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1532098bb09448c5a561b53a029108c3 in term 1.
I20260928 08:35:33.331578 3445 raft_consensus.cc:2454] T e83d606dad3c4429a76dac1d522194b9 P 84ba6c788b2f4d0093e50c745a4f3e19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1532098bb09448c5a561b53a029108c3 in term 1.
I20260928 08:35:33.331962 3642 leader_election.cc:302] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1532098bb09448c5a561b53a029108c3, 43203538ba614070bdbc04cf8a0585c3; no voters:
W20260928 08:35:33.332309 3752 tablet.cc:2369] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:35:33.332378 3777 raft_consensus.cc:2781] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:35:33.332955 3777 raft_consensus.cc:686] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3 [term 1 LEADER]: Becoming Leader. State: Replica: 1532098bb09448c5a561b53a029108c3, State: Running, Role: LEADER
I20260928 08:35:33.333554 3777 consensus_queue.cc:237] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43203538ba614070bdbc04cf8a0585c3" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34421 } } peers { permanent_uuid: "84ba6c788b2f4d0093e50c745a4f3e19" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 35979 } } peers { permanent_uuid: "1532098bb09448c5a561b53a029108c3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 39389 } }
I20260928 08:35:33.338379 3304 catalog_manager.cc:5495] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1532098bb09448c5a561b53a029108c3 (127.15.134.195). New cstate: current_term: 1 leader_uuid: "1532098bb09448c5a561b53a029108c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43203538ba614070bdbc04cf8a0585c3" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84ba6c788b2f4d0093e50c745a4f3e19" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 35979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1532098bb09448c5a561b53a029108c3" member_type: VOTER last_known_addr { host: "127.15.134.195" port: 39389 } health_report { overall_health: HEALTHY } } }
W20260928 08:35:33.368202 3622 tablet.cc:2369] T e83d606dad3c4429a76dac1d522194b9 P 43203538ba614070bdbc04cf8a0585c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:35:33.387063 3491 tablet.cc:2369] T e83d606dad3c4429a76dac1d522194b9 P 84ba6c788b2f4d0093e50c745a4f3e19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:35:33.393100 3767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:33.387403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54742 (local address 127.15.134.195:39389)
0928 08:35:33.387577 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:33.387581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:33.387612 (+ 31us) server_negotiation.cc:407] Connection header received
0928 08:35:33.387695 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:33.387699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:33.387776 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:33.387894 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:35:33.388355 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.389200 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.390755 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.391183 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.391310 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:33.391830 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:35:33.392401 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:35:33.392521 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:33.392753 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 08:35:33.392817 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 08:35:33.401154 3445 raft_consensus.cc:1261] T e83d606dad3c4429a76dac1d522194b9 P 84ba6c788b2f4d0093e50c745a4f3e19 [term 1 FOLLOWER]: Refusing update from remote peer 1532098bb09448c5a561b53a029108c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:35:33.401283 3576 raft_consensus.cc:1261] T e83d606dad3c4429a76dac1d522194b9 P 43203538ba614070bdbc04cf8a0585c3 [term 1 FOLLOWER]: Refusing update from remote peer 1532098bb09448c5a561b53a029108c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:35:33.402199 3779 consensus_queue.cc:1035] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84ba6c788b2f4d0093e50c745a4f3e19" member_type: VOTER last_known_addr { host: "127.15.134.193" port: 35979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:35:33.402554 3777 consensus_queue.cc:1035] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43203538ba614070bdbc04cf8a0585c3" member_type: VOTER last_known_addr { host: "127.15.134.194" port: 34421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:35:33.418260 3784 mvcc.cc:205] Tried to move back new op lower bound from 7334234248801128448 to 7334234248538439680. Current Snapshot: MvccSnapshot[applied={T|T < 7334234248801128448}]
I20260928 08:35:33.432351 3303 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:35:33.432621 3303 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:35:33.436373 3303 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc9ceb63cbf545829000ff02f14441c7: Sending DeleteTablet for 3 replicas of tablet e83d606dad3c4429a76dac1d522194b9
I20260928 08:35:33.437685 3556 tablet_service.cc:1514] Processing DeleteTablet for tablet e83d606dad3c4429a76dac1d522194b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:35:33 UTC) from {username='slave'} at 127.0.0.1:58260
I20260928 08:35:33.437664 3425 tablet_service.cc:1514] Processing DeleteTablet for tablet e83d606dad3c4429a76dac1d522194b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:35:33 UTC) from {username='slave'} at 127.0.0.1:46410
I20260928 08:35:33.438632 3793 tablet_replica.cc:330] T e83d606dad3c4429a76dac1d522194b9 P 43203538ba614070bdbc04cf8a0585c3: stopping tablet replica
I20260928 08:35:33.438580 3686 tablet_service.cc:1514] Processing DeleteTablet for tablet e83d606dad3c4429a76dac1d522194b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:35:33 UTC) from {username='slave'} at 127.0.0.1:54734
I20260928 08:35:33.439265 3794 tablet_replica.cc:330] T e83d606dad3c4429a76dac1d522194b9 P 84ba6c788b2f4d0093e50c745a4f3e19: stopping tablet replica
I20260928 08:35:33.439404 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 3363
I20260928 08:35:33.439919 3793 raft_consensus.cc:2229] T e83d606dad3c4429a76dac1d522194b9 P 43203538ba614070bdbc04cf8a0585c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:35:33.440433 3794 raft_consensus.cc:2229] T e83d606dad3c4429a76dac1d522194b9 P 84ba6c788b2f4d0093e50c745a4f3e19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:35:33.441135 3793 raft_consensus.cc:2258] T e83d606dad3c4429a76dac1d522194b9 P 43203538ba614070bdbc04cf8a0585c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:35:33.441665 3794 raft_consensus.cc:2258] T e83d606dad3c4429a76dac1d522194b9 P 84ba6c788b2f4d0093e50c745a4f3e19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:35:33.444134 3793 ts_tablet_manager.cc:1905] T e83d606dad3c4429a76dac1d522194b9 P 43203538ba614070bdbc04cf8a0585c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:35:33.445616 3794 ts_tablet_manager.cc:1905] T e83d606dad3c4429a76dac1d522194b9 P 84ba6c788b2f4d0093e50c745a4f3e19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:35:33.456671 3793 ts_tablet_manager.cc:1918] T e83d606dad3c4429a76dac1d522194b9 P 43203538ba614070bdbc04cf8a0585c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:35:33.457921 3793 log.cc:1196] T e83d606dad3c4429a76dac1d522194b9 P 43203538ba614070bdbc04cf8a0585c3: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-1/wal/wals/e83d606dad3c4429a76dac1d522194b9
I20260928 08:35:33.458133 3795 tablet_replica.cc:330] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3: stopping tablet replica
I20260928 08:35:33.458515 3793 ts_tablet_manager.cc:1939] T e83d606dad3c4429a76dac1d522194b9 P 43203538ba614070bdbc04cf8a0585c3: Deleting consensus metadata
I20260928 08:35:33.458629 3795 raft_consensus.cc:2229] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:35:33.459981 3795 raft_consensus.cc:2258] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:35:33.460405 3291 catalog_manager.cc:4841] TS 43203538ba614070bdbc04cf8a0585c3 (127.15.134.194:34421): tablet e83d606dad3c4429a76dac1d522194b9 (table test-table [id=5c9db0de29b547bb946ad85ffa0fa183]) successfully deleted
I20260928 08:35:33.462126 3794 ts_tablet_manager.cc:1918] T e83d606dad3c4429a76dac1d522194b9 P 84ba6c788b2f4d0093e50c745a4f3e19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:35:33.462371 3794 log.cc:1196] T e83d606dad3c4429a76dac1d522194b9 P 84ba6c788b2f4d0093e50c745a4f3e19: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-0/wal/wals/e83d606dad3c4429a76dac1d522194b9
I20260928 08:35:33.462564 3795 ts_tablet_manager.cc:1905] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:35:33.463035 3794 ts_tablet_manager.cc:1939] T e83d606dad3c4429a76dac1d522194b9 P 84ba6c788b2f4d0093e50c745a4f3e19: Deleting consensus metadata
I20260928 08:35:33.465373 3290 catalog_manager.cc:4841] TS 84ba6c788b2f4d0093e50c745a4f3e19 (127.15.134.193:35979): tablet e83d606dad3c4429a76dac1d522194b9 (table test-table [id=5c9db0de29b547bb946ad85ffa0fa183]) successfully deleted
I20260928 08:35:33.468986 3795 ts_tablet_manager.cc:1918] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:35:33.469063 3766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:33.442422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46414 (local address 127.15.134.193:35979)
0928 08:35:33.442627 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:33.442634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:33.451644 (+ 9010us) server_negotiation.cc:407] Connection header received
0928 08:35:33.452224 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:33.452234 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:33.452388 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:33.452483 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:33.453407 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.454904 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.458195 (+ 3291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.458885 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.459010 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:33.467137 (+ 8127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:33.467150 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:33.467168 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:33.468016 (+ 848us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:33.468087 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:33.468114 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:33.468240 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:33.468491 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 08:35:33.468656 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 08:35:33.469563 3795 log.cc:1196] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3: Deleting WAL directory at /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790584205252592-15899-0/minicluster-data/ts-2/wal/wals/e83d606dad3c4429a76dac1d522194b9
I20260928 08:35:33.469780 3485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:33.470273 3795 ts_tablet_manager.cc:1939] T e83d606dad3c4429a76dac1d522194b9 P 1532098bb09448c5a561b53a029108c3: Deleting consensus metadata
I20260928 08:35:33.473608 3291 catalog_manager.cc:4841] TS 1532098bb09448c5a561b53a029108c3 (127.15.134.195:39389): tablet e83d606dad3c4429a76dac1d522194b9 (table test-table [id=5c9db0de29b547bb946ad85ffa0fa183]) successfully deleted
I20260928 08:35:33.571045 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 3363
I20260928 08:35:33.590634 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 3494
I20260928 08:35:33.598691 3765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:33.591551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58266 (local address 127.15.134.194:34421)
0928 08:35:33.591769 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:33.591775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:33.591804 (+ 29us) server_negotiation.cc:407] Connection header received
0928 08:35:33.591874 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:33.591881 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:33.592001 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:33.592115 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:33.592492 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.593683 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.594934 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.595620 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.595748 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:33.597068 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:33.597085 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:33.597106 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:33.597886 (+ 780us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:33.597946 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:33.597970 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:33.598086 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:33.598169 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 08:35:33.598317 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 08:35:33.599491 3616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:33.690999 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 3494
I20260928 08:35:33.710647 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 3624
I20260928 08:35:33.718711 3767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:33.711619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54750 (local address 127.15.134.195:39389)
0928 08:35:33.711806 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:33.711811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:33.711829 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:35:33.711923 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:33.711927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:33.712003 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:33.712085 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:33.712598 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.713447 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.715357 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:33.715816 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:33.715929 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:33.717291 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:33.717302 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:33.717317 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:33.717903 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:33.717954 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:33.717978 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:33.718061 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:33.718235 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 08:35:33.718319 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 08:35:33.719511 3746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:33.842257 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 3624
I20260928 08:35:33.862758 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 3273
I20260928 08:35:33.863924 3334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:33.997700 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 3273
2026-09-28T08:35:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2986 ms)
[----------] 6 tests from EncryptionPolicyTest (22663 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:35:34 dist-test-slave-qp2n krb5kdc[3803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:35:34 dist-test-slave-qp2n krb5kdc[3803](info): set up 2 sockets
Sep 28 08:35:34 dist-test-slave-qp2n krb5kdc[3803](info): commencing operation
krb5kdc: starting...
W20260928 08:35:35.138633 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.004s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:35:35 dist-test-slave-qp2n krb5kdc[3803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:35:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:35:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:35:35.407843 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:35:35.753571 3817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:35.753892 3817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:35.753988 3817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:35.761252 3817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:35:35.761373 3817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:35.761420 3817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:35.761483 3817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:35:35.761528 3817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:35:35.772639 3817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46053
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:35.774015 3817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:35.775293 3817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:35.788283 3822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:35.788287 3823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:35.788946 3825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:35.789692 3817 server_base.cc:1034] running on GCE node
I20260928 08:35:35.790763 3817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:35.792321 3817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:35.793583 3817 hybrid_clock.cc:648] HybridClock initialized: now 1790584535793516 us; error 60 us; skew 500 ppm
I20260928 08:35:35.794010 3817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:35:35 dist-test-slave-qp2n krb5kdc[3803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:35:35.797431 3817 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:35:35.799108 3817 webserver.cc:460] Webserver started at http://127.0.0.1:42857/ using document root <none> and password file <none>
I20260928 08:35:35.799849 3817 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:35.800036 3817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:35.800364 3817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:35.803577 3817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "d5c8694befb54878ae875fefd0cc2ff8"
format_stamp: "Formatted at 2026-09-28 08:35:35 on dist-test-slave-qp2n"
server_key: "7ea21b7e634f54168c07439d66e48c9d"
server_key_iv: "5a364d7525d5d8aa242bff8a014f2209"
server_key_version: "encryptionkey@0"
I20260928 08:35:35.804392 3817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "d5c8694befb54878ae875fefd0cc2ff8"
format_stamp: "Formatted at 2026-09-28 08:35:35 on dist-test-slave-qp2n"
server_key: "7ea21b7e634f54168c07439d66e48c9d"
server_key_iv: "5a364d7525d5d8aa242bff8a014f2209"
server_key_version: "encryptionkey@0"
I20260928 08:35:35.809887 3817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260928 08:35:35.813546 3832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:35.815052 3817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:35:35.815266 3817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "d5c8694befb54878ae875fefd0cc2ff8"
format_stamp: "Formatted at 2026-09-28 08:35:35 on dist-test-slave-qp2n"
server_key: "7ea21b7e634f54168c07439d66e48c9d"
server_key_iv: "5a364d7525d5d8aa242bff8a014f2209"
server_key_version: "encryptionkey@0"
I20260928 08:35:35.815493 3817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:35.848035 3817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:35.852312 3817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:35.852648 3817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:35.872411 3817 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46053
I20260928 08:35:35.872466 3883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46053 every 8 connection(s)
I20260928 08:35:35.873874 3817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:35:35.877754 3884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:35.883432 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 3817
I20260928 08:35:35.883656 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:35:35.884335 15899 external_mini_cluster.cc:1428] Setting key 5488315449657e3ca62d69b74ccea6b7
I20260928 08:35:35.889761 3884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5c8694befb54878ae875fefd0cc2ff8: Bootstrap starting.
I20260928 08:35:35.893735 3884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5c8694befb54878ae875fefd0cc2ff8: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:35.895002 3884 log.cc:824] T 00000000000000000000000000000000 P d5c8694befb54878ae875fefd0cc2ff8: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:35.898463 3884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5c8694befb54878ae875fefd0cc2ff8: No bootstrap required, opened a new log
Sep 28 08:35:35 dist-test-slave-qp2n krb5kdc[3803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 08:35:35.907677 3884 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5c8694befb54878ae875fefd0cc2ff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c8694befb54878ae875fefd0cc2ff8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46053 } }
I20260928 08:35:35.908171 3884 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5c8694befb54878ae875fefd0cc2ff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:35.908304 3884 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5c8694befb54878ae875fefd0cc2ff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c8694befb54878ae875fefd0cc2ff8, State: Initialized, Role: FOLLOWER
I20260928 08:35:35.909039 3884 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5c8694befb54878ae875fefd0cc2ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c8694befb54878ae875fefd0cc2ff8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46053 } }
I20260928 08:35:35.909313 3884 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5c8694befb54878ae875fefd0cc2ff8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:35:35.909492 3884 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5c8694befb54878ae875fefd0cc2ff8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:35:35.909691 3884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5c8694befb54878ae875fefd0cc2ff8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:35.910293 3887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:35.886586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44047 (local address 127.0.0.1:46053)
0928 08:35:35.887490 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:35.887505 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:35.887558 (+ 53us) server_negotiation.cc:407] Connection header received
0928 08:35:35.890033 (+ 2475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:35.890065 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:35.890625 (+ 560us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:35.891093 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:35.892538 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:35.894623 (+ 2085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:35.896254 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:35.897367 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:35.897557 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:35.901934 (+ 4377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:35.901942 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:35.901957 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:35.901997 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:35.905292 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:35.906270 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:35.906277 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:35.906287 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:35.906490 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:35.906995 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:35.906998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:35.907000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:35.907566 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:35.907801 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:35.908556 (+ 755us) server_negotiation.cc:299] Negotiation successful
0928 08:35:35.908981 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":426,"threads_started":1}
I20260928 08:35:35.912220 3884 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5c8694befb54878ae875fefd0cc2ff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c8694befb54878ae875fefd0cc2ff8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46053 } }
I20260928 08:35:35.912847 3884 leader_election.cc:302] T 00000000000000000000000000000000 P d5c8694befb54878ae875fefd0cc2ff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5c8694befb54878ae875fefd0cc2ff8; no voters:
I20260928 08:35:35.913573 3884 leader_election.cc:288] T 00000000000000000000000000000000 P d5c8694befb54878ae875fefd0cc2ff8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:35:35.913717 3889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5c8694befb54878ae875fefd0cc2ff8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:35:35.914937 3889 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5c8694befb54878ae875fefd0cc2ff8 [term 1 LEADER]: Becoming Leader. State: Replica: d5c8694befb54878ae875fefd0cc2ff8, State: Running, Role: LEADER
I20260928 08:35:35.915120 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:35:35.915589 3889 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5c8694befb54878ae875fefd0cc2ff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c8694befb54878ae875fefd0cc2ff8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46053 } }
I20260928 08:35:35.915866 3884 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5c8694befb54878ae875fefd0cc2ff8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:35:35.926913 3891 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5c8694befb54878ae875fefd0cc2ff8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5c8694befb54878ae875fefd0cc2ff8. Latest consensus state: current_term: 1 leader_uuid: "d5c8694befb54878ae875fefd0cc2ff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c8694befb54878ae875fefd0cc2ff8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46053 } } }
I20260928 08:35:35.927420 3891 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5c8694befb54878ae875fefd0cc2ff8 [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:35.928021 3890 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5c8694befb54878ae875fefd0cc2ff8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5c8694befb54878ae875fefd0cc2ff8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5c8694befb54878ae875fefd0cc2ff8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46053 } } }
I20260928 08:35:35.928331 3890 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5c8694befb54878ae875fefd0cc2ff8 [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:35.938313 3904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:35:35.950199 3904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:35:35.957248 3887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:35.928340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38655 (local address 127.0.0.1:46053)
0928 08:35:35.928572 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:35.928578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:35.935588 (+ 7010us) server_negotiation.cc:407] Connection header received
0928 08:35:35.936186 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:35.936195 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:35.936310 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:35.936399 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:35.938730 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:35.939926 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:35.942432 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:35.943389 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:35.943523 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:35.947951 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:35.947961 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:35.947966 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:35.948015 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:35.952706 (+ 4691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:35.953848 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:35.953863 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:35.953866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:35.954014 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:35.954598 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:35.954613 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:35.954617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:35.955063 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:35.955361 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:35.956602 (+ 1241us) server_negotiation.cc:299] Negotiation successful
0928 08:35:35.956839 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 08:35:35.964215 3904 catalog_manager.cc:1347] Generated new cluster ID: 1d053fb9f3314a50a979608f3232e8e3
I20260928 08:35:35.964385 3904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:35:35.985273 3904 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:35:35.987195 3904 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:35:35.995815 3904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5c8694befb54878ae875fefd0cc2ff8: Generated new TSK 0
I20260928 08:35:35.996842 3904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:35:36.009318 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 3817
I20260928 08:35:36.011240 3879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:36.105113 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 3817
W20260928 08:35:36.123989 3801 connection.cc:535] client connection to 127.0.0.1:46053 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46053): Connection reset by peer (error 104)
2026-09-28T08:35:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2098 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:35:36 dist-test-slave-qp2n krb5kdc[3916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:35:36 dist-test-slave-qp2n krb5kdc[3916](info): set up 2 sockets
Sep 28 08:35:36 dist-test-slave-qp2n krb5kdc[3916](info): commencing operation
krb5kdc: starting...
W20260928 08:35:37.230551 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:35:37 dist-test-slave-qp2n krb5kdc[3916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:35:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:35:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:35:37.496840 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42243
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:35:37.841310 3929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:37.841655 3929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:37.841835 3929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:37.848996 3929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:35:37.849148 3929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:37.849261 3929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:37.849375 3929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:35:37.849463 3929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:35:37.859308 3929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42243
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:37.860659 3929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:37.862017 3929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:37.874426 3937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:37.874506 3934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:37.874532 3935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:37.875470 3929 server_base.cc:1034] running on GCE node
I20260928 08:35:37.876308 3929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:37.877957 3929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:37.879168 3929 hybrid_clock.cc:648] HybridClock initialized: now 1790584537879118 us; error 46 us; skew 500 ppm
I20260928 08:35:37.879554 3929 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:35:37 dist-test-slave-qp2n krb5kdc[3916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:35:37.883116 3929 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:35:37.884846 3929 webserver.cc:460] Webserver started at http://127.0.0.1:39399/ using document root <none> and password file <none>
I20260928 08:35:37.885584 3929 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:37.885689 3929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:37.886008 3929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:37.889348 3929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "af66ffc556ca40d1aeacdea17d973372"
format_stamp: "Formatted at 2026-09-28 08:35:37 on dist-test-slave-qp2n"
server_key: "0aca335c2ec9f7fb178883026d36f42c"
server_key_iv: "6491546cd1129c82f669559418a6b70a"
server_key_version: "encryptionkey@0"
I20260928 08:35:37.890236 3929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "af66ffc556ca40d1aeacdea17d973372"
format_stamp: "Formatted at 2026-09-28 08:35:37 on dist-test-slave-qp2n"
server_key: "0aca335c2ec9f7fb178883026d36f42c"
server_key_iv: "6491546cd1129c82f669559418a6b70a"
server_key_version: "encryptionkey@0"
I20260928 08:35:37.896180 3929 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 08:35:37.899993 3944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:37.901530 3929 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:35:37.901737 3929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "af66ffc556ca40d1aeacdea17d973372"
format_stamp: "Formatted at 2026-09-28 08:35:37 on dist-test-slave-qp2n"
server_key: "0aca335c2ec9f7fb178883026d36f42c"
server_key_iv: "6491546cd1129c82f669559418a6b70a"
server_key_version: "encryptionkey@0"
I20260928 08:35:37.901985 3929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:37.930917 3929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:37.936197 3929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:37.936558 3929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:37.957171 3929 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42243
I20260928 08:35:37.957204 3995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42243 every 8 connection(s)
I20260928 08:35:37.958595 3929 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:35:37.962513 3996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:37.963908 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 3929
I20260928 08:35:37.964118 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:35:37.964582 15899 external_mini_cluster.cc:1428] Setting key 20e0197604e3ddd13da2a928471cde06
I20260928 08:35:37.976898 3996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af66ffc556ca40d1aeacdea17d973372: Bootstrap starting.
I20260928 08:35:37.978534 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:37.966365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60763 (local address 127.0.0.1:42243)
0928 08:35:37.967157 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:37.967169 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:37.967210 (+ 41us) server_negotiation.cc:407] Connection header received
0928 08:35:37.968837 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:37.968863 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:37.969272 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:37.969594 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:37.970217 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:37.971563 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:37.973623 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:37.974398 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:37.974563 (+ 165us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:37.975974 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:37.975982 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:37.975998 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:37.976565 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:37.976716 (+ 151us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:37.976745 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:37.976879 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:37.977249 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 08:35:37.977469 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":372,"threads_started":1}
I20260928 08:35:37.980340 3996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af66ffc556ca40d1aeacdea17d973372: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:37.981602 3996 log.cc:824] T 00000000000000000000000000000000 P af66ffc556ca40d1aeacdea17d973372: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:37.982607 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:35:37.984711 3996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af66ffc556ca40d1aeacdea17d973372: No bootstrap required, opened a new log
I20260928 08:35:37.994654 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:37.987356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53157 (local address 127.0.0.1:42243)
0928 08:35:37.987576 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:37.987582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:37.987926 (+ 344us) server_negotiation.cc:407] Connection header received
0928 08:35:37.988092 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:37.988095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:37.988164 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:37.988267 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:37.988950 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:37.989761 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:37.991807 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:37.992294 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:37.992410 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:37.993766 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:37.993777 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:37.993781 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:37.993806 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:37.993877 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:37.993897 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:37.993991 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:37.994209 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 08:35:37.994328 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 08:35:37.994974 3996 raft_consensus.cc:348] T 00000000000000000000000000000000 P af66ffc556ca40d1aeacdea17d973372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af66ffc556ca40d1aeacdea17d973372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42243 } }
I20260928 08:35:37.995543 3996 raft_consensus.cc:374] T 00000000000000000000000000000000 P af66ffc556ca40d1aeacdea17d973372 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:37.995675 3996 raft_consensus.cc:729] T 00000000000000000000000000000000 P af66ffc556ca40d1aeacdea17d973372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af66ffc556ca40d1aeacdea17d973372, State: Initialized, Role: FOLLOWER
I20260928 08:35:37.996415 3996 consensus_queue.cc:260] T 00000000000000000000000000000000 P af66ffc556ca40d1aeacdea17d973372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af66ffc556ca40d1aeacdea17d973372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42243 } }
I20260928 08:35:37.996730 3996 raft_consensus.cc:388] T 00000000000000000000000000000000 P af66ffc556ca40d1aeacdea17d973372 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:35:37.996876 3996 raft_consensus.cc:482] T 00000000000000000000000000000000 P af66ffc556ca40d1aeacdea17d973372 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:35:37.997040 3996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af66ffc556ca40d1aeacdea17d973372 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:37.999583 3996 raft_consensus.cc:504] T 00000000000000000000000000000000 P af66ffc556ca40d1aeacdea17d973372 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af66ffc556ca40d1aeacdea17d973372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42243 } }
I20260928 08:35:38.000161 3996 leader_election.cc:302] T 00000000000000000000000000000000 P af66ffc556ca40d1aeacdea17d973372 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af66ffc556ca40d1aeacdea17d973372; no voters:
I20260928 08:35:38.000784 3996 leader_election.cc:288] T 00000000000000000000000000000000 P af66ffc556ca40d1aeacdea17d973372 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:35:38.000953 4006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af66ffc556ca40d1aeacdea17d973372 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:35:38.001654 4006 raft_consensus.cc:686] T 00000000000000000000000000000000 P af66ffc556ca40d1aeacdea17d973372 [term 1 LEADER]: Becoming Leader. State: Replica: af66ffc556ca40d1aeacdea17d973372, State: Running, Role: LEADER
I20260928 08:35:38.002187 4006 consensus_queue.cc:237] T 00000000000000000000000000000000 P af66ffc556ca40d1aeacdea17d973372 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af66ffc556ca40d1aeacdea17d973372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42243 } }
I20260928 08:35:38.002970 3996 sys_catalog.cc:564] T 00000000000000000000000000000000 P af66ffc556ca40d1aeacdea17d973372 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:35:38.005373 4008 sys_catalog.cc:455] T 00000000000000000000000000000000 P af66ffc556ca40d1aeacdea17d973372 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af66ffc556ca40d1aeacdea17d973372. Latest consensus state: current_term: 1 leader_uuid: "af66ffc556ca40d1aeacdea17d973372" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af66ffc556ca40d1aeacdea17d973372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42243 } } }
I20260928 08:35:38.005872 4008 sys_catalog.cc:458] T 00000000000000000000000000000000 P af66ffc556ca40d1aeacdea17d973372 [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:38.011097 4007 sys_catalog.cc:455] T 00000000000000000000000000000000 P af66ffc556ca40d1aeacdea17d973372 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af66ffc556ca40d1aeacdea17d973372" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af66ffc556ca40d1aeacdea17d973372" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42243 } } }
I20260928 08:35:38.011463 4007 sys_catalog.cc:458] T 00000000000000000000000000000000 P af66ffc556ca40d1aeacdea17d973372 [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:38.012243 4013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:35:38.018306 4013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:35:38.028008 4013 catalog_manager.cc:1347] Generated new cluster ID: 20e1a617e3bd4b1c9b71e120f0aeca84
I20260928 08:35:38.028167 4013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:35:38.049221 4013 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:35:38.050616 4013 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:35:38.059801 4013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af66ffc556ca40d1aeacdea17d973372: Generated new TSK 0
I20260928 08:35:38.060786 4013 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:35:38.068895 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 3929
W20260928 08:35:38.070083 3991 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60763
W20260928 08:35:38.070911 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 3929: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:35:38.071096 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 3929
W20260928 08:35:38.091076 3914 connection.cc:535] client connection to 127.0.0.1:42243 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42243): Connection reset by peer (error 104)
2026-09-28T08:35:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1967 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:35:38 dist-test-slave-qp2n krb5kdc[4027](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:35:38 dist-test-slave-qp2n krb5kdc[4027](info): set up 2 sockets
Sep 28 08:35:38 dist-test-slave-qp2n krb5kdc[4027](info): commencing operation
krb5kdc: starting...
W20260928 08:35:39.190686 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:35:39 dist-test-slave-qp2n krb5kdc[4027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:35:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:35:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:35:39.460968 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36043
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:35:39.817297 4040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:39.817629 4040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:39.817847 4040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:39.826078 4040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:35:39.826201 4040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:39.826298 4040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:39.826397 4040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:35:39.826467 4040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:35:39.837149 4040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36043
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:39.838636 4040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:39.840103 4040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:39.853086 4045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:39.853145 4046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:39.853063 4048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:39.854084 4040 server_base.cc:1034] running on GCE node
I20260928 08:35:39.854986 4040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:39.856600 4040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:39.857933 4040 hybrid_clock.cc:648] HybridClock initialized: now 1790584539857862 us; error 76 us; skew 500 ppm
I20260928 08:35:39.858318 4040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:35:39 dist-test-slave-qp2n krb5kdc[4027](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:35:39.867455 4040 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:35:39.869236 4040 webserver.cc:460] Webserver started at http://127.0.0.1:34743/ using document root <none> and password file <none>
I20260928 08:35:39.869987 4040 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:39.870138 4040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:39.870451 4040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:39.873641 4040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "68a262f540fa40a7b9d54a29f7cddf78"
format_stamp: "Formatted at 2026-09-28 08:35:39 on dist-test-slave-qp2n"
server_key: "6f3c0e7041e515835210f431dc484b5c"
server_key_iv: "8b33a69454cd857594802452f739c164"
server_key_version: "encryptionkey@0"
I20260928 08:35:39.874455 4040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "68a262f540fa40a7b9d54a29f7cddf78"
format_stamp: "Formatted at 2026-09-28 08:35:39 on dist-test-slave-qp2n"
server_key: "6f3c0e7041e515835210f431dc484b5c"
server_key_iv: "8b33a69454cd857594802452f739c164"
server_key_version: "encryptionkey@0"
I20260928 08:35:39.879783 4040 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 08:35:39.883517 4055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:39.885085 4040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:35:39.885281 4040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "68a262f540fa40a7b9d54a29f7cddf78"
format_stamp: "Formatted at 2026-09-28 08:35:39 on dist-test-slave-qp2n"
server_key: "6f3c0e7041e515835210f431dc484b5c"
server_key_iv: "8b33a69454cd857594802452f739c164"
server_key_version: "encryptionkey@0"
I20260928 08:35:39.885492 4040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:39.891499 4040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:39.896106 4040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:39.896425 4040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:39.916836 4040 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36043
I20260928 08:35:39.916939 4106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36043 every 8 connection(s)
I20260928 08:35:39.918279 4040 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:35:39.922434 4107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:39.927636 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 4040
I20260928 08:35:39.927814 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:35:39.928253 15899 external_mini_cluster.cc:1428] Setting key 4516245a6bcf3fa9783ade1bf6626176
I20260928 08:35:39.933530 4107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68a262f540fa40a7b9d54a29f7cddf78: Bootstrap starting.
I20260928 08:35:39.936278 4110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:39.930152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45177 (local address 127.0.0.1:36043)
0928 08:35:39.931001 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:39.931013 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:39.931053 (+ 40us) server_negotiation.cc:407] Connection header received
0928 08:35:39.932826 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:39.932857 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:39.933208 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:39.933588 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:39.934677 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:39.934684 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:39.934697 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:39.934710 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:39.934770 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:39.934789 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:39.935012 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:39.935110 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 08:35:39.935314 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":408,"threads_started":1}
I20260928 08:35:39.937407 4107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68a262f540fa40a7b9d54a29f7cddf78: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:39.938514 4107 log.cc:824] T 00000000000000000000000000000000 P 68a262f540fa40a7b9d54a29f7cddf78: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:39.940441 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:35:39.941910 4107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68a262f540fa40a7b9d54a29f7cddf78: No bootstrap required, opened a new log
I20260928 08:35:39.948184 4110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:39.945839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35545 (local address 127.0.0.1:36043)
0928 08:35:39.946044 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:39.946050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:39.946319 (+ 269us) server_negotiation.cc:407] Connection header received
0928 08:35:39.946460 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:39.946466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:39.946531 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:39.946649 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:39.947542 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:39.947548 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:39.947553 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:39.947571 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:39.947625 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:39.947646 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:39.947777 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:39.947827 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 08:35:39.947887 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 08:35:39.951754 4107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68a262f540fa40a7b9d54a29f7cddf78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68a262f540fa40a7b9d54a29f7cddf78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36043 } }
I20260928 08:35:39.952163 4107 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68a262f540fa40a7b9d54a29f7cddf78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:39.952245 4107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68a262f540fa40a7b9d54a29f7cddf78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a262f540fa40a7b9d54a29f7cddf78, State: Initialized, Role: FOLLOWER
I20260928 08:35:39.952804 4107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68a262f540fa40a7b9d54a29f7cddf78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68a262f540fa40a7b9d54a29f7cddf78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36043 } }
I20260928 08:35:39.953018 4107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68a262f540fa40a7b9d54a29f7cddf78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:35:39.953147 4107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68a262f540fa40a7b9d54a29f7cddf78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:35:39.953341 4107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68a262f540fa40a7b9d54a29f7cddf78 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:39.955762 4107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68a262f540fa40a7b9d54a29f7cddf78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68a262f540fa40a7b9d54a29f7cddf78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36043 } }
I20260928 08:35:39.956226 4107 leader_election.cc:302] T 00000000000000000000000000000000 P 68a262f540fa40a7b9d54a29f7cddf78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68a262f540fa40a7b9d54a29f7cddf78; no voters:
I20260928 08:35:39.956914 4107 leader_election.cc:288] T 00000000000000000000000000000000 P 68a262f540fa40a7b9d54a29f7cddf78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:35:39.957063 4117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68a262f540fa40a7b9d54a29f7cddf78 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:35:39.957741 4117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68a262f540fa40a7b9d54a29f7cddf78 [term 1 LEADER]: Becoming Leader. State: Replica: 68a262f540fa40a7b9d54a29f7cddf78, State: Running, Role: LEADER
I20260928 08:35:39.958271 4117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68a262f540fa40a7b9d54a29f7cddf78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68a262f540fa40a7b9d54a29f7cddf78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36043 } }
I20260928 08:35:39.959014 4107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68a262f540fa40a7b9d54a29f7cddf78 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:35:39.961864 4118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68a262f540fa40a7b9d54a29f7cddf78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68a262f540fa40a7b9d54a29f7cddf78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68a262f540fa40a7b9d54a29f7cddf78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36043 } } }
I20260928 08:35:39.962258 4118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68a262f540fa40a7b9d54a29f7cddf78 [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:39.962716 4119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68a262f540fa40a7b9d54a29f7cddf78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68a262f540fa40a7b9d54a29f7cddf78. Latest consensus state: current_term: 1 leader_uuid: "68a262f540fa40a7b9d54a29f7cddf78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68a262f540fa40a7b9d54a29f7cddf78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36043 } } }
I20260928 08:35:39.963126 4119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68a262f540fa40a7b9d54a29f7cddf78 [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:39.964179 4124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:35:39.970194 4124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:35:39.979596 4124 catalog_manager.cc:1347] Generated new cluster ID: b819e298862b4e889f145ee09a8d892f
I20260928 08:35:39.979815 4124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:35:39.992950 4124 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:35:39.993449 4124 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:35:40.003217 4124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68a262f540fa40a7b9d54a29f7cddf78: Generated new TSK 0
I20260928 08:35:40.004204 4124 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:35:40.012029 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 4040
W20260928 08:35:40.013432 4102 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45177
W20260928 08:35:40.014272 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 4040: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:35:40.014452 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 4040
2026-09-28T08:35:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1938 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:35:40 dist-test-slave-qp2n krb5kdc[4138](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:35:40 dist-test-slave-qp2n krb5kdc[4138](info): set up 2 sockets
Sep 28 08:35:40 dist-test-slave-qp2n krb5kdc[4138](info): commencing operation
krb5kdc: starting...
W20260928 08:35:41.126588 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:35:41 dist-test-slave-qp2n krb5kdc[4138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:35:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:35:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:35:41.396950 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33131
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:35:41.745534 4151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:41.745906 4151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:41.746066 4151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:41.753763 4151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:35:41.753939 4151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:41.754042 4151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:41.754148 4151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:35:41.754223 4151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:35:41.764570 4151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:41.766067 4151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:41.767387 4151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:41.779584 4156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:41.779654 4157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:41.779872 4159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:41.780689 4151 server_base.cc:1034] running on GCE node
I20260928 08:35:41.781466 4151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:41.782922 4151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:41.784118 4151 hybrid_clock.cc:648] HybridClock initialized: now 1790584541784079 us; error 30 us; skew 500 ppm
I20260928 08:35:41.784473 4151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:35:41 dist-test-slave-qp2n krb5kdc[4138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:35:41.787633 4151 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:35:41.789245 4151 webserver.cc:460] Webserver started at http://127.0.0.1:36743/ using document root <none> and password file <none>
I20260928 08:35:41.789952 4151 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:41.790108 4151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:41.790424 4151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:41.793623 4151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "94be50133a934b5596c7219adde944c9"
format_stamp: "Formatted at 2026-09-28 08:35:41 on dist-test-slave-qp2n"
server_key: "4eeda67e219c7fcbd23e46e0eb7619a1"
server_key_iv: "58455ce39b5a40330bcf993433b467e0"
server_key_version: "encryptionkey@0"
I20260928 08:35:41.794448 4151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "94be50133a934b5596c7219adde944c9"
format_stamp: "Formatted at 2026-09-28 08:35:41 on dist-test-slave-qp2n"
server_key: "4eeda67e219c7fcbd23e46e0eb7619a1"
server_key_iv: "58455ce39b5a40330bcf993433b467e0"
server_key_version: "encryptionkey@0"
I20260928 08:35:41.800189 4151 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:35:41.803815 4166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:41.805337 4151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:35:41.805497 4151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "94be50133a934b5596c7219adde944c9"
format_stamp: "Formatted at 2026-09-28 08:35:41 on dist-test-slave-qp2n"
server_key: "4eeda67e219c7fcbd23e46e0eb7619a1"
server_key_iv: "58455ce39b5a40330bcf993433b467e0"
server_key_version: "encryptionkey@0"
I20260928 08:35:41.805677 4151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:41.823030 4151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:41.827036 4151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:41.827266 4151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:41.846474 4151 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33131
I20260928 08:35:41.846472 4217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33131 every 8 connection(s)
I20260928 08:35:41.847971 4151 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:35:41.852084 4218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:41.852919 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 4151
I20260928 08:35:41.853089 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:35:41.853549 15899 external_mini_cluster.cc:1428] Setting key 64c78c540bb655e1f8146ccac15c338b
I20260928 08:35:41.867724 4218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94be50133a934b5596c7219adde944c9: Bootstrap starting.
Sep 28 08:35:41 dist-test-slave-qp2n krb5kdc[4138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 08:35:41.871546 4218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94be50133a934b5596c7219adde944c9: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:41.873255 4218 log.cc:824] T 00000000000000000000000000000000 P 94be50133a934b5596c7219adde944c9: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:41.876974 4218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94be50133a934b5596c7219adde944c9: No bootstrap required, opened a new log
I20260928 08:35:41.880302 4221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:41.855548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54399 (local address 127.0.0.1:33131)
0928 08:35:41.856329 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:41.856346 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:41.856413 (+ 67us) server_negotiation.cc:407] Connection header received
0928 08:35:41.859060 (+ 2647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:41.859089 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:41.859634 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:41.860109 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:41.861626 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:41.863806 (+ 2180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:41.865300 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:41.866422 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:41.866616 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:41.870921 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:41.870934 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:41.870954 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:41.871016 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:41.875237 (+ 4221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:41.875982 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:41.875993 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:41.876004 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:41.876257 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:41.876729 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:41.876734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:41.876738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:41.877553 (+ 815us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:41.877912 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:41.878312 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 08:35:41.878800 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":312,"threads_started":1}
I20260928 08:35:41.884358 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:35:41.887302 4218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94be50133a934b5596c7219adde944c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94be50133a934b5596c7219adde944c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33131 } }
I20260928 08:35:41.887848 4218 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94be50133a934b5596c7219adde944c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:41.887981 4218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94be50133a934b5596c7219adde944c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94be50133a934b5596c7219adde944c9, State: Initialized, Role: FOLLOWER
I20260928 08:35:41.888748 4218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94be50133a934b5596c7219adde944c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94be50133a934b5596c7219adde944c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33131 } }
I20260928 08:35:41.889196 4218 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94be50133a934b5596c7219adde944c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:35:41.889364 4218 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94be50133a934b5596c7219adde944c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:35:41.889578 4218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94be50133a934b5596c7219adde944c9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:41.892199 4218 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94be50133a934b5596c7219adde944c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94be50133a934b5596c7219adde944c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33131 } }
I20260928 08:35:41.893164 4218 leader_election.cc:302] T 00000000000000000000000000000000 P 94be50133a934b5596c7219adde944c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94be50133a934b5596c7219adde944c9; no voters:
I20260928 08:35:41.893992 4218 leader_election.cc:288] T 00000000000000000000000000000000 P 94be50133a934b5596c7219adde944c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:35:41.894057 4229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94be50133a934b5596c7219adde944c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:35:41.895570 4229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94be50133a934b5596c7219adde944c9 [term 1 LEADER]: Becoming Leader. State: Replica: 94be50133a934b5596c7219adde944c9, State: Running, Role: LEADER
I20260928 08:35:41.896245 4229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94be50133a934b5596c7219adde944c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94be50133a934b5596c7219adde944c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33131 } }
I20260928 08:35:41.897493 4218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94be50133a934b5596c7219adde944c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:35:41.899439 4231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94be50133a934b5596c7219adde944c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94be50133a934b5596c7219adde944c9. Latest consensus state: current_term: 1 leader_uuid: "94be50133a934b5596c7219adde944c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94be50133a934b5596c7219adde944c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33131 } } }
I20260928 08:35:41.899986 4231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94be50133a934b5596c7219adde944c9 [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:41.900309 4230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94be50133a934b5596c7219adde944c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94be50133a934b5596c7219adde944c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94be50133a934b5596c7219adde944c9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33131 } } }
I20260928 08:35:41.903549 4230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94be50133a934b5596c7219adde944c9 [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:41.907693 4221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:41.889002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55921 (local address 127.0.0.1:33131)
0928 08:35:41.889216 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:41.889225 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:41.889489 (+ 264us) server_negotiation.cc:407] Connection header received
0928 08:35:41.889693 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:41.889701 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:41.889790 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:41.889913 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:41.891396 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:41.892101 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:41.893472 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:41.894279 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:41.894436 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:41.897857 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:41.897864 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:41.897869 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:41.897906 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:41.903027 (+ 5121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:41.904141 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:41.904150 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:41.904157 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:41.904523 (+ 366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:41.905238 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:41.905246 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:41.905253 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:41.905871 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:41.906352 (+ 481us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:41.906741 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 08:35:41.907073 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 08:35:41.910188 4234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:35:41.915774 4234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:35:41.926172 4234 catalog_manager.cc:1347] Generated new cluster ID: a2df90a4ad344102b341a8b8b9771f65
I20260928 08:35:41.926323 4234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:35:41.937217 4234 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:35:41.938583 4234 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:35:41.944567 4234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94be50133a934b5596c7219adde944c9: Generated new TSK 0
I20260928 08:35:41.945709 4234 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:35:41.962186 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 4151
I20260928 08:35:41.963400 4213 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:42.055516 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 4151
2026-09-28T08:35:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2044 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:35:42 dist-test-slave-qp2n krb5kdc[4252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:35:42 dist-test-slave-qp2n krb5kdc[4252](info): set up 2 sockets
Sep 28 08:35:42 dist-test-slave-qp2n krb5kdc[4252](info): commencing operation
krb5kdc: starting...
W20260928 08:35:43.186434 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:35:43 dist-test-slave-qp2n krb5kdc[4252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:35:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:35:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:35:43.461037 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39697
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:35:43.805639 4265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:43.805982 4265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:43.806077 4265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:43.813783 4265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:35:43.813918 4265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:43.813964 4265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:43.814024 4265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:35:43.814067 4265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:35:43.824007 4265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:43.825307 4265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:43.826592 4265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:43.838964 4270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:43.839051 4273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:43.839018 4271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:43.840406 4265 server_base.cc:1034] running on GCE node
I20260928 08:35:43.841351 4265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:43.843182 4265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:43.844483 4265 hybrid_clock.cc:648] HybridClock initialized: now 1790584543844449 us; error 52 us; skew 500 ppm
I20260928 08:35:43.844933 4265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:35:43 dist-test-slave-qp2n krb5kdc[4252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:35:43.852125 4265 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:35:43.853916 4265 webserver.cc:460] Webserver started at http://127.0.0.1:36611/ using document root <none> and password file <none>
I20260928 08:35:43.854635 4265 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:43.854781 4265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:43.855093 4265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:43.858314 4265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "bd618cc967f0418a9405ae0b49f768b5"
format_stamp: "Formatted at 2026-09-28 08:35:43 on dist-test-slave-qp2n"
server_key: "e600e79b7e75076d2f1a4db8528412bd"
server_key_iv: "b76dfb3131a54198fc62fc1268b0110c"
server_key_version: "encryptionkey@0"
I20260928 08:35:43.859124 4265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "bd618cc967f0418a9405ae0b49f768b5"
format_stamp: "Formatted at 2026-09-28 08:35:43 on dist-test-slave-qp2n"
server_key: "e600e79b7e75076d2f1a4db8528412bd"
server_key_iv: "b76dfb3131a54198fc62fc1268b0110c"
server_key_version: "encryptionkey@0"
I20260928 08:35:43.864362 4265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 08:35:43.868019 4280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:43.869567 4265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:35:43.869762 4265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "bd618cc967f0418a9405ae0b49f768b5"
format_stamp: "Formatted at 2026-09-28 08:35:43 on dist-test-slave-qp2n"
server_key: "e600e79b7e75076d2f1a4db8528412bd"
server_key_iv: "b76dfb3131a54198fc62fc1268b0110c"
server_key_version: "encryptionkey@0"
I20260928 08:35:43.869964 4265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:43.893798 4265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:43.898144 4265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:43.898450 4265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:43.917338 4265 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39697
I20260928 08:35:43.917380 4331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39697 every 8 connection(s)
I20260928 08:35:43.918646 4265 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:35:43.918998 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 4265
I20260928 08:35:43.919171 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:35:43.919734 15899 external_mini_cluster.cc:1428] Setting key cc2acdb1545f2d470530679278ae3897
I20260928 08:35:43.923064 4332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:43.935599 4336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:43.921524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60439 (local address 127.0.0.1:39697)
0928 08:35:43.922285 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:43.922296 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:43.922337 (+ 41us) server_negotiation.cc:407] Connection header received
0928 08:35:43.924057 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:43.924080 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:43.924463 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:43.924831 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:43.925476 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:43.926878 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:43.928771 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:43.929820 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:43.929979 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:43.931562 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:43.931574 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:43.931592 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:43.932329 (+ 737us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:43.932442 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:43.932473 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:43.932574 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:43.933519 (+ 945us) server_negotiation.cc:299] Negotiation successful
0928 08:35:43.933919 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":356,"threads_started":1}
I20260928 08:35:43.938128 4332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd618cc967f0418a9405ae0b49f768b5: Bootstrap starting.
I20260928 08:35:43.941133 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:35:43.941987 4332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd618cc967f0418a9405ae0b49f768b5: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:43.943011 4332 log.cc:824] T 00000000000000000000000000000000 P bd618cc967f0418a9405ae0b49f768b5: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:43.946874 4332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd618cc967f0418a9405ae0b49f768b5: No bootstrap required, opened a new log
I20260928 08:35:43.959172 4332 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd618cc967f0418a9405ae0b49f768b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd618cc967f0418a9405ae0b49f768b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39697 } }
I20260928 08:35:43.959832 4332 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd618cc967f0418a9405ae0b49f768b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:43.959944 4332 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd618cc967f0418a9405ae0b49f768b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd618cc967f0418a9405ae0b49f768b5, State: Initialized, Role: FOLLOWER
I20260928 08:35:43.960736 4332 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd618cc967f0418a9405ae0b49f768b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd618cc967f0418a9405ae0b49f768b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39697 } }
I20260928 08:35:43.961037 4332 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd618cc967f0418a9405ae0b49f768b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:35:43.961169 4332 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd618cc967f0418a9405ae0b49f768b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:35:43.961349 4332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd618cc967f0418a9405ae0b49f768b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:43.961601 4336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:43.947612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46403 (local address 127.0.0.1:39697)
0928 08:35:43.947832 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:43.947840 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:43.950422 (+ 2582us) server_negotiation.cc:407] Connection header received
0928 08:35:43.951150 (+ 728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:43.951162 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:43.951258 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:43.951382 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:43.952190 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:43.953587 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:43.955784 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:43.956753 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:43.956907 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:43.960777 (+ 3870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:43.960783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:43.960786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:43.960807 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:43.960854 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:43.960871 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:43.960982 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:43.961111 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 08:35:43.961218 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:35:43.964443 4332 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd618cc967f0418a9405ae0b49f768b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd618cc967f0418a9405ae0b49f768b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39697 } }
I20260928 08:35:43.965138 4332 leader_election.cc:302] T 00000000000000000000000000000000 P bd618cc967f0418a9405ae0b49f768b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd618cc967f0418a9405ae0b49f768b5; no voters:
I20260928 08:35:43.965920 4332 leader_election.cc:288] T 00000000000000000000000000000000 P bd618cc967f0418a9405ae0b49f768b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:35:43.966048 4343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd618cc967f0418a9405ae0b49f768b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:35:43.966818 4343 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd618cc967f0418a9405ae0b49f768b5 [term 1 LEADER]: Becoming Leader. State: Replica: bd618cc967f0418a9405ae0b49f768b5, State: Running, Role: LEADER
I20260928 08:35:43.967434 4343 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd618cc967f0418a9405ae0b49f768b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd618cc967f0418a9405ae0b49f768b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39697 } }
I20260928 08:35:43.968505 4332 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd618cc967f0418a9405ae0b49f768b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:35:43.970736 4344 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd618cc967f0418a9405ae0b49f768b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd618cc967f0418a9405ae0b49f768b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd618cc967f0418a9405ae0b49f768b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39697 } } }
I20260928 08:35:43.971225 4344 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd618cc967f0418a9405ae0b49f768b5 [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:43.979934 4345 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd618cc967f0418a9405ae0b49f768b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd618cc967f0418a9405ae0b49f768b5. Latest consensus state: current_term: 1 leader_uuid: "bd618cc967f0418a9405ae0b49f768b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd618cc967f0418a9405ae0b49f768b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39697 } } }
I20260928 08:35:43.980357 4345 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd618cc967f0418a9405ae0b49f768b5 [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:43.981302 4351 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:35:43.987614 4351 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:35:43.997081 4351 catalog_manager.cc:1347] Generated new cluster ID: f7e59a954dca4353b5d6e6168f31a13a
I20260928 08:35:43.997208 4351 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:35:44.029137 4351 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:35:44.030673 4351 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:35:44.039625 4351 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd618cc967f0418a9405ae0b49f768b5: Generated new TSK 0
I20260928 08:35:44.040495 4351 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:35:44.050298 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 4265
W20260928 08:35:44.051553 4327 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60439
W20260928 08:35:44.052382 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 4265: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:35:44.052551 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 4265
2026-09-28T08:35:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1995 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:35:44 dist-test-slave-qp2n krb5kdc[4364](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:35:44 dist-test-slave-qp2n krb5kdc[4364](info): set up 2 sockets
Sep 28 08:35:44 dist-test-slave-qp2n krb5kdc[4364](info): commencing operation
krb5kdc: starting...
W20260928 08:35:45.187322 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:35:45 dist-test-slave-qp2n krb5kdc[4364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:35:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:35:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:35:45.464226 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35541
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:35:45.812842 4377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:45.813180 4377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:45.813355 4377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:45.821110 4377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:35:45.821233 4377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:45.821346 4377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:45.821483 4377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:35:45.821561 4377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:35:45.832713 4377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:45.834115 4377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:45.835572 4377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:45.849112 4382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:45.848886 4383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:45.849069 4385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:45.849709 4377 server_base.cc:1034] running on GCE node
I20260928 08:35:45.850606 4377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:45.852156 4377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:45.853444 4377 hybrid_clock.cc:648] HybridClock initialized: now 1790584545853388 us; error 50 us; skew 500 ppm
I20260928 08:35:45.853937 4377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:35:45 dist-test-slave-qp2n krb5kdc[4364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:35:45.862229 4377 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:35:45.864094 4377 webserver.cc:460] Webserver started at http://127.0.0.1:34511/ using document root <none> and password file <none>
I20260928 08:35:45.865252 4377 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:45.865427 4377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:45.865794 4377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:45.869195 4377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "cf80ae24e2554045bf213b35a0d8b1ba"
format_stamp: "Formatted at 2026-09-28 08:35:45 on dist-test-slave-qp2n"
server_key: "9f12078fef9eee5b7a6bb45e4d35a8b2"
server_key_iv: "4d30dd1d0be624278d0d0569fb7b5095"
server_key_version: "encryptionkey@0"
I20260928 08:35:45.870118 4377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "cf80ae24e2554045bf213b35a0d8b1ba"
format_stamp: "Formatted at 2026-09-28 08:35:45 on dist-test-slave-qp2n"
server_key: "9f12078fef9eee5b7a6bb45e4d35a8b2"
server_key_iv: "4d30dd1d0be624278d0d0569fb7b5095"
server_key_version: "encryptionkey@0"
I20260928 08:35:45.875924 4377 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:35:45.879787 4392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:45.881446 4377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 08:35:45.881665 4377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "cf80ae24e2554045bf213b35a0d8b1ba"
format_stamp: "Formatted at 2026-09-28 08:35:45 on dist-test-slave-qp2n"
server_key: "9f12078fef9eee5b7a6bb45e4d35a8b2"
server_key_iv: "4d30dd1d0be624278d0d0569fb7b5095"
server_key_version: "encryptionkey@0"
I20260928 08:35:45.881937 4377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:45.888486 4377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:45.893370 4377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:45.893731 4377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:45.913074 4377 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35541
I20260928 08:35:45.913092 4443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35541 every 8 connection(s)
I20260928 08:35:45.914636 4377 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:35:45.918596 4444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:45.921170 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 4377
I20260928 08:35:45.921358 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:35:45.921888 15899 external_mini_cluster.cc:1428] Setting key b5382da5c5b4c47150419e74671f8298
I20260928 08:35:45.930114 4447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:45.923988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49673 (local address 127.0.0.1:35541)
0928 08:35:45.924832 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:45.924844 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:45.924889 (+ 45us) server_negotiation.cc:407] Connection header received
0928 08:35:45.926578 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:45.926600 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:45.926944 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:45.927338 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:45.928414 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:45.928420 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:45.928436 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:45.928458 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:45.928515 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:45.928535 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:45.928827 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:45.928973 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 08:35:45.929170 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":355,"threads_started":1}
I20260928 08:35:45.931473 4444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf80ae24e2554045bf213b35a0d8b1ba: Bootstrap starting.
I20260928 08:35:45.934067 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:35:45.934985 4444 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf80ae24e2554045bf213b35a0d8b1ba: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:45.936029 4444 log.cc:824] T 00000000000000000000000000000000 P cf80ae24e2554045bf213b35a0d8b1ba: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:45.939126 4444 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf80ae24e2554045bf213b35a0d8b1ba: No bootstrap required, opened a new log
I20260928 08:35:45.941701 4447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:45.938760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52645 (local address 127.0.0.1:35541)
0928 08:35:45.938944 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:45.938952 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:45.939416 (+ 464us) server_negotiation.cc:407] Connection header received
0928 08:35:45.939566 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:45.939572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:45.939645 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:45.939767 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:45.940960 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:45.940966 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:45.940971 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:45.940990 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:45.941044 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:45.941070 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:45.941210 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:45.941338 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 08:35:45.941401 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 08:35:45.948089 4444 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf80ae24e2554045bf213b35a0d8b1ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf80ae24e2554045bf213b35a0d8b1ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35541 } }
I20260928 08:35:45.948467 4444 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf80ae24e2554045bf213b35a0d8b1ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:45.948611 4444 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf80ae24e2554045bf213b35a0d8b1ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf80ae24e2554045bf213b35a0d8b1ba, State: Initialized, Role: FOLLOWER
I20260928 08:35:45.949339 4444 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf80ae24e2554045bf213b35a0d8b1ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf80ae24e2554045bf213b35a0d8b1ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35541 } }
I20260928 08:35:45.949600 4444 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf80ae24e2554045bf213b35a0d8b1ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:35:45.949787 4444 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf80ae24e2554045bf213b35a0d8b1ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:35:45.949970 4444 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf80ae24e2554045bf213b35a0d8b1ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:45.952414 4444 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf80ae24e2554045bf213b35a0d8b1ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf80ae24e2554045bf213b35a0d8b1ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35541 } }
I20260928 08:35:45.953065 4444 leader_election.cc:302] T 00000000000000000000000000000000 P cf80ae24e2554045bf213b35a0d8b1ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf80ae24e2554045bf213b35a0d8b1ba; no voters:
I20260928 08:35:45.953712 4444 leader_election.cc:288] T 00000000000000000000000000000000 P cf80ae24e2554045bf213b35a0d8b1ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:35:45.953893 4454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf80ae24e2554045bf213b35a0d8b1ba [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:35:45.954625 4454 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf80ae24e2554045bf213b35a0d8b1ba [term 1 LEADER]: Becoming Leader. State: Replica: cf80ae24e2554045bf213b35a0d8b1ba, State: Running, Role: LEADER
I20260928 08:35:45.955242 4454 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf80ae24e2554045bf213b35a0d8b1ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf80ae24e2554045bf213b35a0d8b1ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35541 } }
I20260928 08:35:45.956017 4444 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf80ae24e2554045bf213b35a0d8b1ba [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:35:45.958220 4456 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf80ae24e2554045bf213b35a0d8b1ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf80ae24e2554045bf213b35a0d8b1ba. Latest consensus state: current_term: 1 leader_uuid: "cf80ae24e2554045bf213b35a0d8b1ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf80ae24e2554045bf213b35a0d8b1ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35541 } } }
I20260928 08:35:45.958549 4455 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf80ae24e2554045bf213b35a0d8b1ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf80ae24e2554045bf213b35a0d8b1ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf80ae24e2554045bf213b35a0d8b1ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35541 } } }
I20260928 08:35:45.958854 4456 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf80ae24e2554045bf213b35a0d8b1ba [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:45.958848 4455 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf80ae24e2554045bf213b35a0d8b1ba [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:45.962144 4460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:35:45.967948 4460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:35:45.977309 4460 catalog_manager.cc:1347] Generated new cluster ID: abca823997f14f97bea567af1828733b
I20260928 08:35:45.977440 4460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:35:46.000150 4460 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:35:46.000706 4460 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:35:46.006606 4460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf80ae24e2554045bf213b35a0d8b1ba: Generated new TSK 0
I20260928 08:35:46.007526 4460 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:35:46.024852 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 4377
W20260928 08:35:46.026693 4439 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49673
W20260928 08:35:46.027452 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 4377: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:35:46.027608 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 4377
2026-09-28T08:35:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1974 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:35:46 dist-test-slave-qp2n krb5kdc[4476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:35:46 dist-test-slave-qp2n krb5kdc[4476](info): set up 2 sockets
Sep 28 08:35:46 dist-test-slave-qp2n krb5kdc[4476](info): commencing operation
krb5kdc: starting...
W20260928 08:35:47.150502 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:35:47 dist-test-slave-qp2n krb5kdc[4476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:35:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:35:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:35:47.421564 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45725
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:35:47.778980 4489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:47.779290 4489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:47.779389 4489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:47.786734 4489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:35:47.786857 4489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:47.786906 4489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:47.786967 4489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:35:47.787004 4489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:35:47.797215 4489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:47.798535 4489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:47.799819 4489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:47.812412 4494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:47.812464 4495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:47.812495 4497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:47.813740 4489 server_base.cc:1034] running on GCE node
I20260928 08:35:47.814528 4489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:47.815922 4489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:47.817149 4489 hybrid_clock.cc:648] HybridClock initialized: now 1790584547817102 us; error 69 us; skew 500 ppm
I20260928 08:35:47.817517 4489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:35:47 dist-test-slave-qp2n krb5kdc[4476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:35:47.821013 4489 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:35:47.822475 4489 webserver.cc:460] Webserver started at http://127.0.0.1:41487/ using document root <none> and password file <none>
I20260928 08:35:47.823151 4489 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:47.823256 4489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:47.823585 4489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:47.826570 4489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "49f49b6b295f47ceb5713cec7b236bbc"
format_stamp: "Formatted at 2026-09-28 08:35:47 on dist-test-slave-qp2n"
server_key: "7ea308979b7b6a3693db6880f52d4669"
server_key_iv: "7422cb48ddcd00788c1698f3a984f583"
server_key_version: "encryptionkey@0"
I20260928 08:35:47.827432 4489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "49f49b6b295f47ceb5713cec7b236bbc"
format_stamp: "Formatted at 2026-09-28 08:35:47 on dist-test-slave-qp2n"
server_key: "7ea308979b7b6a3693db6880f52d4669"
server_key_iv: "7422cb48ddcd00788c1698f3a984f583"
server_key_version: "encryptionkey@0"
I20260928 08:35:47.832844 4489 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 08:35:47.836382 4504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:47.837920 4489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:35:47.838111 4489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "49f49b6b295f47ceb5713cec7b236bbc"
format_stamp: "Formatted at 2026-09-28 08:35:47 on dist-test-slave-qp2n"
server_key: "7ea308979b7b6a3693db6880f52d4669"
server_key_iv: "7422cb48ddcd00788c1698f3a984f583"
server_key_version: "encryptionkey@0"
I20260928 08:35:47.838348 4489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:47.853343 4489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:47.857777 4489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:47.858083 4489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:47.877508 4489 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45725
I20260928 08:35:47.877540 4555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45725 every 8 connection(s)
I20260928 08:35:47.878945 4489 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:35:47.882929 4556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:47.888129 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 4489
I20260928 08:35:47.888383 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:35:47.888936 15899 external_mini_cluster.cc:1428] Setting key 548922bdb151401cb9f142aadf076c43
I20260928 08:35:47.894657 4556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f49b6b295f47ceb5713cec7b236bbc: Bootstrap starting.
I20260928 08:35:47.898540 4556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49f49b6b295f47ceb5713cec7b236bbc: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:47.899868 4556 log.cc:824] T 00000000000000000000000000000000 P 49f49b6b295f47ceb5713cec7b236bbc: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:47.903589 4556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49f49b6b295f47ceb5713cec7b236bbc: No bootstrap required, opened a new log
Sep 28 08:35:47 dist-test-slave-qp2n krb5kdc[4476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 08:35:47.912173 4556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49f49b6b295f47ceb5713cec7b236bbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f49b6b295f47ceb5713cec7b236bbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45725 } }
I20260928 08:35:47.912585 4556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49f49b6b295f47ceb5713cec7b236bbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:47.912736 4556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49f49b6b295f47ceb5713cec7b236bbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f49b6b295f47ceb5713cec7b236bbc, State: Initialized, Role: FOLLOWER
I20260928 08:35:47.913523 4556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49f49b6b295f47ceb5713cec7b236bbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f49b6b295f47ceb5713cec7b236bbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45725 } }
I20260928 08:35:47.913805 4556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49f49b6b295f47ceb5713cec7b236bbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:35:47.913981 4556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49f49b6b295f47ceb5713cec7b236bbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:35:47.914173 4556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49f49b6b295f47ceb5713cec7b236bbc [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:47.914739 4559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:47.891075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:54321 (local address 127.0.0.1:45725)
0928 08:35:47.891965 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:47.891980 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:47.892028 (+ 48us) server_negotiation.cc:407] Connection header received
0928 08:35:47.893877 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:47.893903 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:47.894368 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:47.894717 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:47.896516 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:47.897984 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:47.899898 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:47.900956 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:47.901190 (+ 234us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:47.906466 (+ 5276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:47.906475 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:47.906492 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:47.906541 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:47.910062 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:47.910983 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:47.910990 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:47.910997 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:47.911179 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:47.911670 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:47.911679 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:47.911681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:47.912237 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:47.912577 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:47.913189 (+ 612us) server_negotiation.cc:299] Negotiation successful
0928 08:35:47.913527 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":397,"threads_started":1}
I20260928 08:35:47.916592 4556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49f49b6b295f47ceb5713cec7b236bbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f49b6b295f47ceb5713cec7b236bbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45725 } }
I20260928 08:35:47.917214 4556 leader_election.cc:302] T 00000000000000000000000000000000 P 49f49b6b295f47ceb5713cec7b236bbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49f49b6b295f47ceb5713cec7b236bbc; no voters:
I20260928 08:35:47.917907 4556 leader_election.cc:288] T 00000000000000000000000000000000 P 49f49b6b295f47ceb5713cec7b236bbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:35:47.918025 4561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49f49b6b295f47ceb5713cec7b236bbc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:35:47.919137 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:35:47.919976 4556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49f49b6b295f47ceb5713cec7b236bbc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:35:47.920945 4561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49f49b6b295f47ceb5713cec7b236bbc [term 1 LEADER]: Becoming Leader. State: Replica: 49f49b6b295f47ceb5713cec7b236bbc, State: Running, Role: LEADER
I20260928 08:35:47.921446 4561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49f49b6b295f47ceb5713cec7b236bbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f49b6b295f47ceb5713cec7b236bbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45725 } }
I20260928 08:35:47.924120 4563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f49b6b295f47ceb5713cec7b236bbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49f49b6b295f47ceb5713cec7b236bbc. Latest consensus state: current_term: 1 leader_uuid: "49f49b6b295f47ceb5713cec7b236bbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f49b6b295f47ceb5713cec7b236bbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45725 } } }
I20260928 08:35:47.924469 4563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f49b6b295f47ceb5713cec7b236bbc [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:47.923862 4562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49f49b6b295f47ceb5713cec7b236bbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49f49b6b295f47ceb5713cec7b236bbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49f49b6b295f47ceb5713cec7b236bbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45725 } } }
I20260928 08:35:47.925093 4562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49f49b6b295f47ceb5713cec7b236bbc [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:47.931931 4567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:35:47.939863 4567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:35:47.966758 4567 catalog_manager.cc:1347] Generated new cluster ID: 126cc6c0785948dda2cc38d4ddd8ab64
I20260928 08:35:47.966894 4567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:35:47.969102 4559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:47.947517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:52413 (local address 127.0.0.1:45725)
0928 08:35:47.947703 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:47.947713 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:47.947736 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:35:47.947839 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:47.947851 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:47.947952 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:47.948123 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:47.950160 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:47.951377 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:47.953347 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:47.954047 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:47.954151 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:47.958053 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:47.958061 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:47.958067 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:47.958123 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:47.962186 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:47.963455 (+ 1269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:47.963461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:47.963464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:47.963667 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:47.964387 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:47.964396 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:47.964402 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:47.965001 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:47.965356 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:47.966252 (+ 896us) server_negotiation.cc:299] Negotiation successful
0928 08:35:47.966506 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 08:35:47.988646 4567 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:35:47.990574 4567 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:35:47.999393 4567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49f49b6b295f47ceb5713cec7b236bbc: Generated new TSK 0
I20260928 08:35:48.000273 4567 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:35:48.014981 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 4489
I20260928 08:35:48.016589 4551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:48.113399 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 4489
W20260928 08:35:48.128973 4474 connection.cc:535] client connection to 127.0.0.1:45725 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45725): Connection reset by peer (error 104)
2026-09-28T08:35:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2083 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:35:48 dist-test-slave-qp2n krb5kdc[4589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:35:48 dist-test-slave-qp2n krb5kdc[4589](info): set up 2 sockets
Sep 28 08:35:48 dist-test-slave-qp2n krb5kdc[4589](info): commencing operation
krb5kdc: starting...
W20260928 08:35:49.230665 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:35:49 dist-test-slave-qp2n krb5kdc[4589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:35:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:35:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:35:49.505121 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46339
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:35:49.852026 4603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:49.852398 4603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:49.852573 4603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:49.860136 4603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:35:49.860353 4603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:49.860419 4603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:49.860483 4603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:35:49.860520 4603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:35:49.870798 4603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:49.872066 4603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:49.873383 4603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:49.885767 4608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:49.885936 4609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:49.885980 4611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:49.887470 4603 server_base.cc:1034] running on GCE node
I20260928 08:35:49.888401 4603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:49.890071 4603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:49.891335 4603 hybrid_clock.cc:648] HybridClock initialized: now 1790584549891268 us; error 61 us; skew 500 ppm
I20260928 08:35:49.891746 4603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:35:49 dist-test-slave-qp2n krb5kdc[4589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:35:49.895246 4603 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:35:49.896958 4603 webserver.cc:460] Webserver started at http://127.0.0.1:39025/ using document root <none> and password file <none>
I20260928 08:35:49.897684 4603 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:49.897835 4603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:49.898147 4603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:49.901417 4603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "0766b83eb9c2481096a2fef517770f5c"
format_stamp: "Formatted at 2026-09-28 08:35:49 on dist-test-slave-qp2n"
server_key: "3f03c59555562a7dca50b325fc1e920c"
server_key_iv: "0790b9a7197c1bca077bb94dc8b44652"
server_key_version: "encryptionkey@0"
I20260928 08:35:49.902223 4603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "0766b83eb9c2481096a2fef517770f5c"
format_stamp: "Formatted at 2026-09-28 08:35:49 on dist-test-slave-qp2n"
server_key: "3f03c59555562a7dca50b325fc1e920c"
server_key_iv: "0790b9a7197c1bca077bb94dc8b44652"
server_key_version: "encryptionkey@0"
I20260928 08:35:49.907645 4603 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 08:35:49.911265 4618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:49.912772 4603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:35:49.912979 4603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "0766b83eb9c2481096a2fef517770f5c"
format_stamp: "Formatted at 2026-09-28 08:35:49 on dist-test-slave-qp2n"
server_key: "3f03c59555562a7dca50b325fc1e920c"
server_key_iv: "0790b9a7197c1bca077bb94dc8b44652"
server_key_version: "encryptionkey@0"
I20260928 08:35:49.913235 4603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:49.937513 4603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:49.942050 4603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:49.942348 4603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:49.961091 4603 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46339
I20260928 08:35:49.961128 4669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46339 every 8 connection(s)
I20260928 08:35:49.962461 4603 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:35:49.966240 4670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:49.972512 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 4603
I20260928 08:35:49.972743 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:35:49.973310 15899 external_mini_cluster.cc:1428] Setting key 1529efbf7f7c0057e07a990fd634b826
I20260928 08:35:49.978053 4670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0766b83eb9c2481096a2fef517770f5c: Bootstrap starting.
I20260928 08:35:49.981499 4670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0766b83eb9c2481096a2fef517770f5c: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:49.982478 4670 log.cc:824] T 00000000000000000000000000000000 P 0766b83eb9c2481096a2fef517770f5c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:49.985427 4670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0766b83eb9c2481096a2fef517770f5c: No bootstrap required, opened a new log
I20260928 08:35:49.986981 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:49.975156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:35969 (local address 127.0.0.1:46339)
0928 08:35:49.976764 (+ 1608us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:49.976776 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:49.976816 (+ 40us) server_negotiation.cc:407] Connection header received
0928 08:35:49.978542 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:49.978565 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:49.978943 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:49.979323 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:49.979900 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:49.981235 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:49.982606 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:49.983319 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:49.983529 (+ 210us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:49.984802 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:49.984811 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:49.984826 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:49.985351 (+ 525us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:49.985425 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:49.985450 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:49.985613 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:49.985901 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 08:35:49.986068 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1314,"thread_start_us":1072,"threads_started":1}
I20260928 08:35:49.991334 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:35:49.994623 4670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0766b83eb9c2481096a2fef517770f5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0766b83eb9c2481096a2fef517770f5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46339 } }
I20260928 08:35:49.995024 4670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0766b83eb9c2481096a2fef517770f5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:49.995131 4670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0766b83eb9c2481096a2fef517770f5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0766b83eb9c2481096a2fef517770f5c, State: Initialized, Role: FOLLOWER
I20260928 08:35:49.995858 4670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0766b83eb9c2481096a2fef517770f5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0766b83eb9c2481096a2fef517770f5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46339 } }
I20260928 08:35:49.996191 4670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0766b83eb9c2481096a2fef517770f5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:35:49.996353 4670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0766b83eb9c2481096a2fef517770f5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:35:49.996567 4670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0766b83eb9c2481096a2fef517770f5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:49.999691 4670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0766b83eb9c2481096a2fef517770f5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0766b83eb9c2481096a2fef517770f5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46339 } }
I20260928 08:35:50.000335 4670 leader_election.cc:302] T 00000000000000000000000000000000 P 0766b83eb9c2481096a2fef517770f5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0766b83eb9c2481096a2fef517770f5c; no voters:
I20260928 08:35:50.001104 4670 leader_election.cc:288] T 00000000000000000000000000000000 P 0766b83eb9c2481096a2fef517770f5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:35:50.001256 4680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0766b83eb9c2481096a2fef517770f5c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:35:50.003196 4680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0766b83eb9c2481096a2fef517770f5c [term 1 LEADER]: Becoming Leader. State: Replica: 0766b83eb9c2481096a2fef517770f5c, State: Running, Role: LEADER
I20260928 08:35:50.003834 4680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0766b83eb9c2481096a2fef517770f5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0766b83eb9c2481096a2fef517770f5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46339 } }
I20260928 08:35:50.004177 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:49.995794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:47397 (local address 127.0.0.1:46339)
0928 08:35:49.995968 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:49.995972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:49.996566 (+ 594us) server_negotiation.cc:407] Connection header received
0928 08:35:49.996841 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:49.996847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:49.996940 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:49.997053 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:49.997529 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:49.998686 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:50.000115 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:50.000865 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:50.001047 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:50.002712 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:50.002722 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:50.002730 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:50.002763 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:50.002829 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:50.002858 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:50.002999 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:50.003749 (+ 750us) server_negotiation.cc:299] Negotiation successful
0928 08:35:50.003828 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 08:35:50.004175 4670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0766b83eb9c2481096a2fef517770f5c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:35:50.006642 4682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0766b83eb9c2481096a2fef517770f5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0766b83eb9c2481096a2fef517770f5c. Latest consensus state: current_term: 1 leader_uuid: "0766b83eb9c2481096a2fef517770f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0766b83eb9c2481096a2fef517770f5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46339 } } }
I20260928 08:35:50.007025 4682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0766b83eb9c2481096a2fef517770f5c [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:50.007745 4681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0766b83eb9c2481096a2fef517770f5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0766b83eb9c2481096a2fef517770f5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0766b83eb9c2481096a2fef517770f5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46339 } } }
I20260928 08:35:50.008052 4681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0766b83eb9c2481096a2fef517770f5c [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:50.010076 4685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:35:50.016150 4685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:35:50.025674 4685 catalog_manager.cc:1347] Generated new cluster ID: 6b940f1c7e524e81a8c8ed497c5c75bf
I20260928 08:35:50.025797 4685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:35:50.041155 4685 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:35:50.042600 4685 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:35:50.050704 4685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0766b83eb9c2481096a2fef517770f5c: Generated new TSK 0
I20260928 08:35:50.051821 4685 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:35:50.061713 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 4603
W20260928 08:35:50.063715 4665 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.32:35969
W20260928 08:35:50.064702 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 4603: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:35:50.064901 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 4603
W20260928 08:35:50.083670 4587 connection.cc:535] client connection to 127.0.0.1:46339 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46339): Connection reset by peer (error 104)
2026-09-28T08:35:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1954 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:35:50 dist-test-slave-qp2n krb5kdc[4701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:35:50 dist-test-slave-qp2n krb5kdc[4701](info): set up 2 sockets
Sep 28 08:35:50 dist-test-slave-qp2n krb5kdc[4701](info): commencing operation
krb5kdc: starting...
W20260928 08:35:51.242660 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:35:51 dist-test-slave-qp2n krb5kdc[4701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:35:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:35:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:35:51.517071 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34377
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:35:51.868584 4715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:51.868985 4715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:51.869148 4715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:51.876592 4715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:35:51.876794 4715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:51.876883 4715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:51.876979 4715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:35:51.877048 4715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:35:51.887030 4715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34377
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:51.888450 4715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:51.889848 4715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:51.902928 4721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:51.903008 4723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:51.903040 4720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:51.904196 4715 server_base.cc:1034] running on GCE node
I20260928 08:35:51.905056 4715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:51.906565 4715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:51.907801 4715 hybrid_clock.cc:648] HybridClock initialized: now 1790584551907730 us; error 35 us; skew 500 ppm
I20260928 08:35:51.908171 4715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:35:51 dist-test-slave-qp2n krb5kdc[4701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584551, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:35:51.915402 4715 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:35:51.917147 4715 webserver.cc:460] Webserver started at http://127.0.0.1:41119/ using document root <none> and password file <none>
I20260928 08:35:51.917869 4715 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:51.918018 4715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:51.918358 4715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:51.921563 4715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "2747ab2a538d448ab5653a10421f5143"
format_stamp: "Formatted at 2026-09-28 08:35:51 on dist-test-slave-qp2n"
server_key: "64154624c61d35bbbcac3d6e88fc8024"
server_key_iv: "0ca7a97b4fa4d6d21b534dc24e0d1e63"
server_key_version: "encryptionkey@0"
I20260928 08:35:51.922364 4715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "2747ab2a538d448ab5653a10421f5143"
format_stamp: "Formatted at 2026-09-28 08:35:51 on dist-test-slave-qp2n"
server_key: "64154624c61d35bbbcac3d6e88fc8024"
server_key_iv: "0ca7a97b4fa4d6d21b534dc24e0d1e63"
server_key_version: "encryptionkey@0"
I20260928 08:35:51.927938 4715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260928 08:35:51.931674 4730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:51.933223 4715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 08:35:51.933427 4715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "2747ab2a538d448ab5653a10421f5143"
format_stamp: "Formatted at 2026-09-28 08:35:51 on dist-test-slave-qp2n"
server_key: "64154624c61d35bbbcac3d6e88fc8024"
server_key_iv: "0ca7a97b4fa4d6d21b534dc24e0d1e63"
server_key_version: "encryptionkey@0"
I20260928 08:35:51.933642 4715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:51.939756 4715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:51.944315 4715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:51.944720 4715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:51.964423 4715 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34377
I20260928 08:35:51.964483 4781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34377 every 8 connection(s)
I20260928 08:35:51.965909 4715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:35:51.969719 4782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:51.974046 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 4715
I20260928 08:35:51.974236 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:35:51.974669 15899 external_mini_cluster.cc:1428] Setting key 4e3f6c0eec371f9196861744a2d6aa0e
I20260928 08:35:51.982780 4782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2747ab2a538d448ab5653a10421f5143: Bootstrap starting.
I20260928 08:35:51.984908 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:51.976749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:42803 (local address 127.0.0.1:34377)
0928 08:35:51.977593 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:51.977609 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:51.977669 (+ 60us) server_negotiation.cc:407] Connection header received
0928 08:35:51.980733 (+ 3064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:51.980768 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:51.981261 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:51.981743 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:51.982788 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:51.982796 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:51.982816 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:51.982835 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:51.982910 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:51.982944 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:51.983175 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:51.983305 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 08:35:51.983521 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":337,"threads_started":1}
I20260928 08:35:51.986716 4782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2747ab2a538d448ab5653a10421f5143: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:51.987983 4782 log.cc:824] T 00000000000000000000000000000000 P 2747ab2a538d448ab5653a10421f5143: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:51.989085 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:35:51.991042 4782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2747ab2a538d448ab5653a10421f5143: No bootstrap required, opened a new log
I20260928 08:35:51.996577 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:51.993802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:42995 (local address 127.0.0.1:34377)
0928 08:35:51.994009 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:51.994015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:51.994457 (+ 442us) server_negotiation.cc:407] Connection header received
0928 08:35:51.994605 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:51.994610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:51.994674 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:51.994798 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:51.995828 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:51.995835 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:51.995840 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:51.995864 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:51.995922 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:51.995943 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:51.996066 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:51.996186 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 08:35:51.996260 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:35:52.000033 4782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2747ab2a538d448ab5653a10421f5143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2747ab2a538d448ab5653a10421f5143" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34377 } }
I20260928 08:35:52.000447 4782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2747ab2a538d448ab5653a10421f5143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:52.000551 4782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2747ab2a538d448ab5653a10421f5143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2747ab2a538d448ab5653a10421f5143, State: Initialized, Role: FOLLOWER
I20260928 08:35:52.001222 4782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2747ab2a538d448ab5653a10421f5143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2747ab2a538d448ab5653a10421f5143" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34377 } }
I20260928 08:35:52.001482 4782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2747ab2a538d448ab5653a10421f5143 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:35:52.001642 4782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2747ab2a538d448ab5653a10421f5143 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:35:52.001828 4782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2747ab2a538d448ab5653a10421f5143 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:52.004202 4782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2747ab2a538d448ab5653a10421f5143 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2747ab2a538d448ab5653a10421f5143" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34377 } }
I20260928 08:35:52.004827 4782 leader_election.cc:302] T 00000000000000000000000000000000 P 2747ab2a538d448ab5653a10421f5143 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2747ab2a538d448ab5653a10421f5143; no voters:
I20260928 08:35:52.005473 4782 leader_election.cc:288] T 00000000000000000000000000000000 P 2747ab2a538d448ab5653a10421f5143 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:35:52.005662 4792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2747ab2a538d448ab5653a10421f5143 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:35:52.006428 4792 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2747ab2a538d448ab5653a10421f5143 [term 1 LEADER]: Becoming Leader. State: Replica: 2747ab2a538d448ab5653a10421f5143, State: Running, Role: LEADER
I20260928 08:35:52.007118 4792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2747ab2a538d448ab5653a10421f5143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2747ab2a538d448ab5653a10421f5143" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34377 } }
I20260928 08:35:52.007697 4782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2747ab2a538d448ab5653a10421f5143 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:35:52.010430 4794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2747ab2a538d448ab5653a10421f5143 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2747ab2a538d448ab5653a10421f5143. Latest consensus state: current_term: 1 leader_uuid: "2747ab2a538d448ab5653a10421f5143" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2747ab2a538d448ab5653a10421f5143" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34377 } } }
I20260928 08:35:52.011008 4794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2747ab2a538d448ab5653a10421f5143 [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:52.010896 4793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2747ab2a538d448ab5653a10421f5143 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2747ab2a538d448ab5653a10421f5143" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2747ab2a538d448ab5653a10421f5143" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34377 } } }
I20260928 08:35:52.011224 4793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2747ab2a538d448ab5653a10421f5143 [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:52.014294 4800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:35:52.020318 4800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:35:52.029872 4800 catalog_manager.cc:1347] Generated new cluster ID: e20a7ba720944c419d2c305a174e2148
I20260928 08:35:52.030020 4800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:35:52.043723 4800 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:35:52.044323 4800 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:35:52.054551 4800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2747ab2a538d448ab5653a10421f5143: Generated new TSK 0
I20260928 08:35:52.055734 4800 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:35:52.067157 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 4715
W20260928 08:35:52.068388 4777 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.32:42803
W20260928 08:35:52.069168 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 4715: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:35:52.069335 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 4715
2026-09-28T08:35:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2003 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:35:52 dist-test-slave-qp2n krb5kdc[4813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:35:52 dist-test-slave-qp2n krb5kdc[4813](info): set up 2 sockets
Sep 28 08:35:52 dist-test-slave-qp2n krb5kdc[4813](info): commencing operation
krb5kdc: starting...
W20260928 08:35:53.218739 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:35:53 dist-test-slave-qp2n krb5kdc[4813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:35:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:35:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:35:53.497124 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43533
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:35:53.848845 4827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:53.849164 4827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:53.849376 4827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:53.857571 4827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:35:53.857746 4827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:53.857856 4827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:53.857977 4827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:35:53.858057 4827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:35:53.868391 4827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43533
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:53.869778 4827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:53.871112 4827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:53.883741 4832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:53.883669 4833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:53.883749 4835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:53.884864 4827 server_base.cc:1034] running on GCE node
I20260928 08:35:53.885751 4827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:53.887228 4827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:53.888471 4827 hybrid_clock.cc:648] HybridClock initialized: now 1790584553888416 us; error 49 us; skew 500 ppm
I20260928 08:35:53.888918 4827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:35:53 dist-test-slave-qp2n krb5kdc[4813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584553, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:35:53.892381 4827 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:35:53.894094 4827 webserver.cc:460] Webserver started at http://127.0.0.1:41817/ using document root <none> and password file <none>
I20260928 08:35:53.894923 4827 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:53.895087 4827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:53.895422 4827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:53.898617 4827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "11a920af2e5744b5b992c4c01b82e3fa"
format_stamp: "Formatted at 2026-09-28 08:35:53 on dist-test-slave-qp2n"
server_key: "6a7cff54953934a31d790f792e9300c9"
server_key_iv: "274912e51d9a2ab9271666cbba24d8c7"
server_key_version: "encryptionkey@0"
I20260928 08:35:53.899487 4827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "11a920af2e5744b5b992c4c01b82e3fa"
format_stamp: "Formatted at 2026-09-28 08:35:53 on dist-test-slave-qp2n"
server_key: "6a7cff54953934a31d790f792e9300c9"
server_key_iv: "274912e51d9a2ab9271666cbba24d8c7"
server_key_version: "encryptionkey@0"
I20260928 08:35:53.905045 4827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 08:35:53.908782 4842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:53.910351 4827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:35:53.910573 4827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "11a920af2e5744b5b992c4c01b82e3fa"
format_stamp: "Formatted at 2026-09-28 08:35:53 on dist-test-slave-qp2n"
server_key: "6a7cff54953934a31d790f792e9300c9"
server_key_iv: "274912e51d9a2ab9271666cbba24d8c7"
server_key_version: "encryptionkey@0"
I20260928 08:35:53.910799 4827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:53.933859 4827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:53.938483 4827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:53.938810 4827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:53.958874 4827 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43533
I20260928 08:35:53.958909 4893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43533 every 8 connection(s)
I20260928 08:35:53.960327 4827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:35:53.963646 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 4827
I20260928 08:35:53.963867 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:35:53.964316 4894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:53.964342 15899 external_mini_cluster.cc:1428] Setting key 4056d57ebf131e893753255304b92ae3
I20260928 08:35:53.979642 4894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11a920af2e5744b5b992c4c01b82e3fa: Bootstrap starting.
Sep 28 08:35:53 dist-test-slave-qp2n krb5kdc[4813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 08:35:53.982949 4894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11a920af2e5744b5b992c4c01b82e3fa: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:53.984030 4894 log.cc:824] T 00000000000000000000000000000000 P 11a920af2e5744b5b992c4c01b82e3fa: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:53.986814 4894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11a920af2e5744b5b992c4c01b82e3fa: No bootstrap required, opened a new log
I20260928 08:35:53.988549 4897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:53.966418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:37543 (local address 127.0.0.1:43533)
0928 08:35:53.967317 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:53.967332 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:53.967374 (+ 42us) server_negotiation.cc:407] Connection header received
0928 08:35:53.969141 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:53.969171 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:53.969595 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:53.969939 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:53.971842 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:53.973279 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:53.975159 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:53.975878 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:53.976022 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:53.981019 (+ 4997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:53.981028 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:53.981043 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:53.981085 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:53.984300 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:53.985206 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:53.985213 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:53.985219 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:53.985406 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:53.985885 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:53.985891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:53.985893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:53.986401 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:53.986731 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:53.987280 (+ 549us) server_negotiation.cc:299] Negotiation successful
0928 08:35:53.987575 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":381,"threads_started":1}
I20260928 08:35:53.996448 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:35:53.998039 4894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11a920af2e5744b5b992c4c01b82e3fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a920af2e5744b5b992c4c01b82e3fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43533 } }
I20260928 08:35:53.998560 4894 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11a920af2e5744b5b992c4c01b82e3fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:53.998682 4894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11a920af2e5744b5b992c4c01b82e3fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11a920af2e5744b5b992c4c01b82e3fa, State: Initialized, Role: FOLLOWER
I20260928 08:35:53.999397 4894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11a920af2e5744b5b992c4c01b82e3fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a920af2e5744b5b992c4c01b82e3fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43533 } }
I20260928 08:35:53.999734 4894 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11a920af2e5744b5b992c4c01b82e3fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:35:53.999904 4894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11a920af2e5744b5b992c4c01b82e3fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:35:54.000135 4894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11a920af2e5744b5b992c4c01b82e3fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:54.003743 4894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11a920af2e5744b5b992c4c01b82e3fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a920af2e5744b5b992c4c01b82e3fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43533 } }
I20260928 08:35:54.004540 4894 leader_election.cc:302] T 00000000000000000000000000000000 P 11a920af2e5744b5b992c4c01b82e3fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11a920af2e5744b5b992c4c01b82e3fa; no voters:
I20260928 08:35:54.005429 4894 leader_election.cc:288] T 00000000000000000000000000000000 P 11a920af2e5744b5b992c4c01b82e3fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:35:54.005514 4904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11a920af2e5744b5b992c4c01b82e3fa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:35:54.008195 4894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11a920af2e5744b5b992c4c01b82e3fa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:35:54.012569 4904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11a920af2e5744b5b992c4c01b82e3fa [term 1 LEADER]: Becoming Leader. State: Replica: 11a920af2e5744b5b992c4c01b82e3fa, State: Running, Role: LEADER
I20260928 08:35:54.013371 4904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11a920af2e5744b5b992c4c01b82e3fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a920af2e5744b5b992c4c01b82e3fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43533 } }
I20260928 08:35:54.016726 4905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11a920af2e5744b5b992c4c01b82e3fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11a920af2e5744b5b992c4c01b82e3fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a920af2e5744b5b992c4c01b82e3fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43533 } } }
I20260928 08:35:54.017130 4905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11a920af2e5744b5b992c4c01b82e3fa [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:54.021186 4906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11a920af2e5744b5b992c4c01b82e3fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11a920af2e5744b5b992c4c01b82e3fa. Latest consensus state: current_term: 1 leader_uuid: "11a920af2e5744b5b992c4c01b82e3fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11a920af2e5744b5b992c4c01b82e3fa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43533 } } }
I20260928 08:35:54.021596 4906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11a920af2e5744b5b992c4c01b82e3fa [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:54.025358 4912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:35:54.028285 4897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:54.003207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:41879 (local address 127.0.0.1:43533)
0928 08:35:54.003387 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:54.003396 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:54.004227 (+ 831us) server_negotiation.cc:407] Connection header received
0928 08:35:54.004436 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:54.004441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:54.004520 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:54.004690 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:54.007358 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:54.008993 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:54.011722 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:54.012587 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:54.012728 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:54.017517 (+ 4789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:54.017528 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:54.017536 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:35:54.017595 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:54.024594 (+ 6999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:54.025789 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:54.025795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:54.025799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:54.025967 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:35:54.026532 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:35:54.026537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:35:54.026540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:35:54.026891 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:54.027213 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:54.027682 (+ 469us) server_negotiation.cc:299] Negotiation successful
0928 08:35:54.027924 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 08:35:54.031282 4912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:35:54.059235 4912 catalog_manager.cc:1347] Generated new cluster ID: 405d15f9d835473d8b6df502f58fc191
I20260928 08:35:54.059424 4912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:35:54.077410 4912 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:35:54.078814 4912 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:35:54.103720 4912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11a920af2e5744b5b992c4c01b82e3fa: Generated new TSK 0
I20260928 08:35:54.104769 4912 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:35:54.113155 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 4827
I20260928 08:35:54.114683 4889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:35:54.210124 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 4827
W20260928 08:35:54.229053 4811 connection.cc:535] client connection to 127.0.0.1:43533 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43533): Connection reset by peer (error 104)
2026-09-28T08:35:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2141 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:35:54 dist-test-slave-qp2n krb5kdc[4928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:35:54 dist-test-slave-qp2n krb5kdc[4928](info): set up 2 sockets
Sep 28 08:35:54 dist-test-slave-qp2n krb5kdc[4928](info): commencing operation
krb5kdc: starting...
W20260928 08:35:55.330691 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:35:55 dist-test-slave-qp2n krb5kdc[4928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:35:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:35:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:35:55.612293 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40983
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:35:55.956984 4941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:55.957276 4941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:55.957437 4941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:55.964869 4941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:35:55.964989 4941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:55.965070 4941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:55.965171 4941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:35:55.965241 4941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:35:55.975487 4941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40983
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:55.976871 4941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:55.978252 4941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:55.990679 4946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:55.990720 4949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:55.990759 4947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:55.991616 4941 server_base.cc:1034] running on GCE node
I20260928 08:35:55.992729 4941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:55.994367 4941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:55.995602 4941 hybrid_clock.cc:648] HybridClock initialized: now 1790584555995551 us; error 49 us; skew 500 ppm
I20260928 08:35:55.996049 4941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:35:55 dist-test-slave-qp2n krb5kdc[4928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584555, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:35:55.999686 4941 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:35:56.001601 4941 webserver.cc:460] Webserver started at http://127.0.0.1:34365/ using document root <none> and password file <none>
I20260928 08:35:56.002372 4941 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:56.002528 4941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:56.002863 4941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:56.006220 4941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "9add28504d744ad7be340b5719066b9e"
format_stamp: "Formatted at 2026-09-28 08:35:55 on dist-test-slave-qp2n"
server_key: "839f145c7ec6e40d925ef103d22f3a1c"
server_key_iv: "b8b64a1a5e00a72baae817bb7596966e"
server_key_version: "encryptionkey@0"
I20260928 08:35:56.007102 4941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "9add28504d744ad7be340b5719066b9e"
format_stamp: "Formatted at 2026-09-28 08:35:55 on dist-test-slave-qp2n"
server_key: "839f145c7ec6e40d925ef103d22f3a1c"
server_key_iv: "b8b64a1a5e00a72baae817bb7596966e"
server_key_version: "encryptionkey@0"
I20260928 08:35:56.013095 4941 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 08:35:56.017279 4956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:56.018922 4941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:35:56.019162 4941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "9add28504d744ad7be340b5719066b9e"
format_stamp: "Formatted at 2026-09-28 08:35:55 on dist-test-slave-qp2n"
server_key: "839f145c7ec6e40d925ef103d22f3a1c"
server_key_iv: "b8b64a1a5e00a72baae817bb7596966e"
server_key_version: "encryptionkey@0"
I20260928 08:35:56.019416 4941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:56.058874 4941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:56.063565 4941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:56.063913 4941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:56.083751 4941 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40983
I20260928 08:35:56.083787 5007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40983 every 8 connection(s)
I20260928 08:35:56.085278 4941 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:35:56.089244 5008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:56.092052 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 4941
I20260928 08:35:56.092237 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:35:56.092829 15899 external_mini_cluster.cc:1428] Setting key a9b53e7654ecce27b874db29f8051036
I20260928 08:35:56.101660 5008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9add28504d744ad7be340b5719066b9e: Bootstrap starting.
I20260928 08:35:56.105391 5008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9add28504d744ad7be340b5719066b9e: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:56.106436 5011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:56.094744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:36367 (local address 127.0.0.1:40983)
0928 08:35:56.095566 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:56.095578 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:56.095621 (+ 43us) server_negotiation.cc:407] Connection header received
0928 08:35:56.097402 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:56.097425 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:56.097832 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:56.098171 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:56.098755 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:56.100148 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:56.101961 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:56.102610 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:56.102733 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:56.104057 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:56.104065 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:56.104079 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:56.104587 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:56.104733 (+ 146us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:56.104760 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:56.104887 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:56.105162 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 08:35:56.105383 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":390,"threads_started":1}
I20260928 08:35:56.106441 5008 log.cc:824] T 00000000000000000000000000000000 P 9add28504d744ad7be340b5719066b9e: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:56.109525 5008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9add28504d744ad7be340b5719066b9e: No bootstrap required, opened a new log
I20260928 08:35:56.110257 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:35:56.118108 5008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9add28504d744ad7be340b5719066b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9add28504d744ad7be340b5719066b9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40983 } }
I20260928 08:35:56.118615 5008 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9add28504d744ad7be340b5719066b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:56.118705 5008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9add28504d744ad7be340b5719066b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9add28504d744ad7be340b5719066b9e, State: Initialized, Role: FOLLOWER
I20260928 08:35:56.119375 5008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9add28504d744ad7be340b5719066b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9add28504d744ad7be340b5719066b9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40983 } }
I20260928 08:35:56.119638 5008 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9add28504d744ad7be340b5719066b9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:35:56.119829 5008 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9add28504d744ad7be340b5719066b9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:35:56.120028 5008 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9add28504d744ad7be340b5719066b9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:56.121840 5011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:56.114680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:37323 (local address 127.0.0.1:40983)
0928 08:35:56.114897 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:56.114904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:56.115222 (+ 318us) server_negotiation.cc:407] Connection header received
0928 08:35:56.115460 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:56.115465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:56.115533 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:56.115748 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:56.116351 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:56.117310 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:56.119177 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:35:56.119657 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:35:56.119789 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:35:56.120983 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:56.120991 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:56.120995 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:56.121013 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:56.121078 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:56.121098 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:56.121264 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:56.121425 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 08:35:56.121491 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 08:35:56.123109 5008 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9add28504d744ad7be340b5719066b9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9add28504d744ad7be340b5719066b9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40983 } }
I20260928 08:35:56.123729 5008 leader_election.cc:302] T 00000000000000000000000000000000 P 9add28504d744ad7be340b5719066b9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9add28504d744ad7be340b5719066b9e; no voters:
I20260928 08:35:56.124320 5008 leader_election.cc:288] T 00000000000000000000000000000000 P 9add28504d744ad7be340b5719066b9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:35:56.124547 5018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9add28504d744ad7be340b5719066b9e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:35:56.125404 5018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9add28504d744ad7be340b5719066b9e [term 1 LEADER]: Becoming Leader. State: Replica: 9add28504d744ad7be340b5719066b9e, State: Running, Role: LEADER
I20260928 08:35:56.125984 5018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9add28504d744ad7be340b5719066b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9add28504d744ad7be340b5719066b9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40983 } }
I20260928 08:35:56.126560 5008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9add28504d744ad7be340b5719066b9e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:35:56.128727 5020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9add28504d744ad7be340b5719066b9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9add28504d744ad7be340b5719066b9e. Latest consensus state: current_term: 1 leader_uuid: "9add28504d744ad7be340b5719066b9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9add28504d744ad7be340b5719066b9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40983 } } }
I20260928 08:35:56.129187 5020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9add28504d744ad7be340b5719066b9e [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:56.129442 5019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9add28504d744ad7be340b5719066b9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9add28504d744ad7be340b5719066b9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9add28504d744ad7be340b5719066b9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40983 } } }
I20260928 08:35:56.129840 5019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9add28504d744ad7be340b5719066b9e [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:56.131268 5023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:35:56.137115 5023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:35:56.146739 5023 catalog_manager.cc:1347] Generated new cluster ID: a136e2f1f61e4df7a561bd29ab619d69
I20260928 08:35:56.146996 5023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:35:56.167887 5023 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:35:56.169942 5023 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:35:56.176394 5023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9add28504d744ad7be340b5719066b9e: Generated new TSK 0
I20260928 08:35:56.177608 5023 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:35:56.193799 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 4941
W20260928 08:35:56.195241 5003 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.32:36367
W20260928 08:35:56.196113 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 4941: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:35:56.196264 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 4941
W20260928 08:35:56.214948 4926 connection.cc:535] client connection to 127.0.0.1:40983 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40983): Connection reset by peer (error 104)
2026-09-28T08:35:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1985 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:35:56 dist-test-slave-qp2n krb5kdc[5039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:35:56 dist-test-slave-qp2n krb5kdc[5039](info): set up 2 sockets
Sep 28 08:35:56 dist-test-slave-qp2n krb5kdc[5039](info): commencing operation
krb5kdc: starting...
W20260928 08:35:57.330824 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:35:57 dist-test-slave-qp2n krb5kdc[5039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:35:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:35:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:35:57.602022 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41489
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:35:57.945770 5053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:57.946091 5053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:57.946271 5053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:57.953646 5053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:35:57.953758 5053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:57.953876 5053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:57.953989 5053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:35:57.954069 5053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:35:57.964112 5053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41489
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:57.965526 5053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:57.966857 5053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:57.979280 5058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:57.979252 5061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:57.979221 5059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:57.980123 5053 server_base.cc:1034] running on GCE node
I20260928 08:35:57.981088 5053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:57.982540 5053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:57.983762 5053 hybrid_clock.cc:648] HybridClock initialized: now 1790584557983726 us; error 35 us; skew 500 ppm
I20260928 08:35:57.984141 5053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:35:57 dist-test-slave-qp2n krb5kdc[5039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584557, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:35:57.991071 5053 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:35:57.992758 5053 webserver.cc:460] Webserver started at http://127.0.0.1:35633/ using document root <none> and password file <none>
I20260928 08:35:57.993510 5053 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:57.993667 5053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:57.993988 5053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:57.997246 5053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "68d5004b3e714a59a0b8ab33aec5d2cd"
format_stamp: "Formatted at 2026-09-28 08:35:57 on dist-test-slave-qp2n"
server_key: "fda6b572f8ab5834f16e44eb6002b543"
server_key_iv: "3c89e15f40f42ecefda6cefc66fa73f7"
server_key_version: "encryptionkey@0"
I20260928 08:35:57.998065 5053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "68d5004b3e714a59a0b8ab33aec5d2cd"
format_stamp: "Formatted at 2026-09-28 08:35:57 on dist-test-slave-qp2n"
server_key: "fda6b572f8ab5834f16e44eb6002b543"
server_key_iv: "3c89e15f40f42ecefda6cefc66fa73f7"
server_key_version: "encryptionkey@0"
I20260928 08:35:58.003641 5053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:35:58.007720 5068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:58.009301 5053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:35:58.009512 5053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "68d5004b3e714a59a0b8ab33aec5d2cd"
format_stamp: "Formatted at 2026-09-28 08:35:57 on dist-test-slave-qp2n"
server_key: "fda6b572f8ab5834f16e44eb6002b543"
server_key_iv: "3c89e15f40f42ecefda6cefc66fa73f7"
server_key_version: "encryptionkey@0"
I20260928 08:35:58.009841 5053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:58.016206 5053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:58.021010 5053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:58.021355 5053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:35:58.041692 5053 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41489
I20260928 08:35:58.041860 5119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41489 every 8 connection(s)
I20260928 08:35:58.043185 5053 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:35:58.047515 5120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:35:58.047907 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 5053
I20260928 08:35:58.048089 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:35:58.048568 15899 external_mini_cluster.cc:1428] Setting key d78c9f58d281721edb446ec14a289f69
I20260928 08:35:58.058914 5123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:58.050481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:48765 (local address 127.0.0.1:41489)
0928 08:35:58.051519 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:58.051537 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:58.051603 (+ 66us) server_negotiation.cc:407] Connection header received
0928 08:35:58.054130 (+ 2527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:58.054165 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:58.054713 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:58.055243 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:58.056485 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:58.056495 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:58.056514 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:58.056536 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:58.056662 (+ 126us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:58.056705 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:58.056984 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:58.057163 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 08:35:58.057430 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":390,"threads_started":1}
I20260928 08:35:58.062885 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:35:58.065400 5120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68d5004b3e714a59a0b8ab33aec5d2cd: Bootstrap starting.
I20260928 08:35:58.068994 5120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68d5004b3e714a59a0b8ab33aec5d2cd: Neither blocks nor log segments found. Creating new log.
I20260928 08:35:58.070034 5123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:35:58.067467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:56083 (local address 127.0.0.1:41489)
0928 08:35:58.067645 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 08:35:58.067652 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:35:58.068014 (+ 362us) server_negotiation.cc:407] Connection header received
0928 08:35:58.068172 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:35:58.068176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:35:58.068243 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:35:58.068361 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:35:58.069378 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:35:58.069385 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:35:58.069395 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:35:58.069418 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:35:58.069475 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:35:58.069503 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:35:58.069620 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 08:35:58.069702 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 08:35:58.069762 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 08:35:58.070482 5120 log.cc:824] T 00000000000000000000000000000000 P 68d5004b3e714a59a0b8ab33aec5d2cd: Log is configured to *not* fsync() on all Append() calls
I20260928 08:35:58.073773 5120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68d5004b3e714a59a0b8ab33aec5d2cd: No bootstrap required, opened a new log
I20260928 08:35:58.081238 5120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68d5004b3e714a59a0b8ab33aec5d2cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d5004b3e714a59a0b8ab33aec5d2cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41489 } }
I20260928 08:35:58.081668 5120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68d5004b3e714a59a0b8ab33aec5d2cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:35:58.081773 5120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68d5004b3e714a59a0b8ab33aec5d2cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68d5004b3e714a59a0b8ab33aec5d2cd, State: Initialized, Role: FOLLOWER
I20260928 08:35:58.082408 5120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68d5004b3e714a59a0b8ab33aec5d2cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d5004b3e714a59a0b8ab33aec5d2cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41489 } }
I20260928 08:35:58.082682 5120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68d5004b3e714a59a0b8ab33aec5d2cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:35:58.082834 5120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68d5004b3e714a59a0b8ab33aec5d2cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:35:58.083020 5120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68d5004b3e714a59a0b8ab33aec5d2cd [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:35:58.085445 5120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68d5004b3e714a59a0b8ab33aec5d2cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d5004b3e714a59a0b8ab33aec5d2cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41489 } }
I20260928 08:35:58.085983 5120 leader_election.cc:302] T 00000000000000000000000000000000 P 68d5004b3e714a59a0b8ab33aec5d2cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68d5004b3e714a59a0b8ab33aec5d2cd; no voters:
I20260928 08:35:58.086745 5120 leader_election.cc:288] T 00000000000000000000000000000000 P 68d5004b3e714a59a0b8ab33aec5d2cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:35:58.086910 5130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68d5004b3e714a59a0b8ab33aec5d2cd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:35:58.087761 5130 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68d5004b3e714a59a0b8ab33aec5d2cd [term 1 LEADER]: Becoming Leader. State: Replica: 68d5004b3e714a59a0b8ab33aec5d2cd, State: Running, Role: LEADER
I20260928 08:35:58.088454 5130 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68d5004b3e714a59a0b8ab33aec5d2cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d5004b3e714a59a0b8ab33aec5d2cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41489 } }
I20260928 08:35:58.089541 5120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68d5004b3e714a59a0b8ab33aec5d2cd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:35:58.091877 5132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68d5004b3e714a59a0b8ab33aec5d2cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68d5004b3e714a59a0b8ab33aec5d2cd. Latest consensus state: current_term: 1 leader_uuid: "68d5004b3e714a59a0b8ab33aec5d2cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d5004b3e714a59a0b8ab33aec5d2cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41489 } } }
I20260928 08:35:58.092368 5132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68d5004b3e714a59a0b8ab33aec5d2cd [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:58.091866 5131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68d5004b3e714a59a0b8ab33aec5d2cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68d5004b3e714a59a0b8ab33aec5d2cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68d5004b3e714a59a0b8ab33aec5d2cd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41489 } } }
I20260928 08:35:58.092830 5131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68d5004b3e714a59a0b8ab33aec5d2cd [sys.catalog]: This master's current role is: LEADER
I20260928 08:35:58.095114 5137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:35:58.101174 5137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:35:58.110895 5137 catalog_manager.cc:1347] Generated new cluster ID: a227c65b6539449a85eeb3f3352468eb
I20260928 08:35:58.111053 5137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:35:58.128685 5137 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:35:58.129196 5137 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:35:58.135703 5137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68d5004b3e714a59a0b8ab33aec5d2cd: Generated new TSK 0
I20260928 08:35:58.136809 5137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:35:58.142537 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 5053
W20260928 08:35:58.143916 5115 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.32:48765
W20260928 08:35:58.144886 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 5053: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:35:58.145179 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 5053
2026-09-28T08:35:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1947 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:35:58 dist-test-slave-qp2n krb5kdc[5151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:35:58 dist-test-slave-qp2n krb5kdc[5151](info): set up 2 sockets
Sep 28 08:35:58 dist-test-slave-qp2n krb5kdc[5151](info): commencing operation
krb5kdc: starting...
W20260928 08:35:59.258559 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:35:59 dist-test-slave-qp2n krb5kdc[5151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:35:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:35:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:35:59.524776 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:45405
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:39319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:45405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:35:59.880184 5164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:35:59.880463 5164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:35:59.880641 5164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:35:59.887629 5164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:35:59.887727 5164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:35:59.887820 5164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:35:59.887923 5164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:35:59.888002 5164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:35:59.897476 5164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:39319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:45405
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:45405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:35:59.898711 5164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:35:59.900046 5164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:35:59.930218 5169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:59.930527 5170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:35:59.930735 5172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:35:59.931314 5164 server_base.cc:1034] running on GCE node
I20260928 08:35:59.932544 5164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:35:59.934286 5164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:35:59.935559 5164 hybrid_clock.cc:648] HybridClock initialized: now 1790584559935526 us; error 66 us; skew 500 ppm
I20260928 08:35:59.935995 5164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:35:59 dist-test-slave-qp2n krb5kdc[5151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584559, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:35:59.939464 5164 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:35:59.941493 5164 webserver.cc:460] Webserver started at http://127.15.134.254:39239/ using document root <none> and password file <none>
I20260928 08:35:59.942278 5164 fs_manager.cc:359] Metadata directory not provided
I20260928 08:35:59.942445 5164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:35:59.942809 5164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:35:59.946460 5164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "2f0dd92d919f41259ff791c2a09075b6"
format_stamp: "Formatted at 2026-09-28 08:35:59 on dist-test-slave-qp2n"
server_key: "23a9a70cc2f8b7cee6b8ff9d0b02adf3"
server_key_iv: "f75be12f000776bb0bed7ec9221bcb84"
server_key_version: "encryptionkey@0"
I20260928 08:35:59.947396 5164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "2f0dd92d919f41259ff791c2a09075b6"
format_stamp: "Formatted at 2026-09-28 08:35:59 on dist-test-slave-qp2n"
server_key: "23a9a70cc2f8b7cee6b8ff9d0b02adf3"
server_key_iv: "f75be12f000776bb0bed7ec9221bcb84"
server_key_version: "encryptionkey@0"
I20260928 08:35:59.954167 5164 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260928 08:35:59.958447 5180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:35:59.960001 5164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 08:35:59.960232 5164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "2f0dd92d919f41259ff791c2a09075b6"
format_stamp: "Formatted at 2026-09-28 08:35:59 on dist-test-slave-qp2n"
server_key: "23a9a70cc2f8b7cee6b8ff9d0b02adf3"
server_key_iv: "f75be12f000776bb0bed7ec9221bcb84"
server_key_version: "encryptionkey@0"
I20260928 08:35:59.960458 5164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:35:59.987178 5164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:35:59.991775 5164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:35:59.992092 5164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:36:00.011209 5164 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:45405
I20260928 08:36:00.011258 5231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:45405 every 8 connection(s)
I20260928 08:36:00.012650 5164 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:36:00.016429 5232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:36:00.022629 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 5164
I20260928 08:36:00.022855 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:36:00.023442 15899 external_mini_cluster.cc:1428] Setting key 09838d26e8d29de4cc92d5b7212887d9
I20260928 08:36:00.027408 5232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f0dd92d919f41259ff791c2a09075b6: Bootstrap starting.
I20260928 08:36:00.031224 5232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f0dd92d919f41259ff791c2a09075b6: Neither blocks nor log segments found. Creating new log.
I20260928 08:36:00.032331 5232 log.cc:824] T 00000000000000000000000000000000 P 2f0dd92d919f41259ff791c2a09075b6: Log is configured to *not* fsync() on all Append() calls
I20260928 08:36:00.035558 5232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f0dd92d919f41259ff791c2a09075b6: No bootstrap required, opened a new log
Sep 28 08:36:00 dist-test-slave-qp2n krb5kdc[5151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:36:00.047128 5232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f0dd92d919f41259ff791c2a09075b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0dd92d919f41259ff791c2a09075b6" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45405 } }
I20260928 08:36:00.047673 5232 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f0dd92d919f41259ff791c2a09075b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:36:00.047807 5232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f0dd92d919f41259ff791c2a09075b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f0dd92d919f41259ff791c2a09075b6, State: Initialized, Role: FOLLOWER
I20260928 08:36:00.048560 5232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f0dd92d919f41259ff791c2a09075b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0dd92d919f41259ff791c2a09075b6" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45405 } }
I20260928 08:36:00.048938 5232 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f0dd92d919f41259ff791c2a09075b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:36:00.049113 5232 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f0dd92d919f41259ff791c2a09075b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:36:00.049338 5232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f0dd92d919f41259ff791c2a09075b6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:36:00.049403 5235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:00.025291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42895 (local address 127.15.134.254:45405)
0928 08:36:00.026092 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:00.026103 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:00.026143 (+ 40us) server_negotiation.cc:407] Connection header received
0928 08:36:00.027922 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:00.027943 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:00.028332 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:00.028726 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:00.030587 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:00.032014 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:00.033702 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:00.034717 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:00.034897 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:00.039110 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:00.039128 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:00.039151 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:00.039210 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:00.043913 (+ 4703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:00.044907 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:00.044918 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:00.044929 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:00.045243 (+ 314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:00.045823 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:00.045831 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:00.045840 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:00.046707 (+ 867us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:00.047046 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:00.047433 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 08:36:00.047831 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":397,"threads_started":1}
I20260928 08:36:00.052101 5232 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f0dd92d919f41259ff791c2a09075b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0dd92d919f41259ff791c2a09075b6" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45405 } }
I20260928 08:36:00.052824 5232 leader_election.cc:302] T 00000000000000000000000000000000 P 2f0dd92d919f41259ff791c2a09075b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f0dd92d919f41259ff791c2a09075b6; no voters:
I20260928 08:36:00.053642 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:36:00.053957 5232 leader_election.cc:288] T 00000000000000000000000000000000 P 2f0dd92d919f41259ff791c2a09075b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:36:00.053993 5237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f0dd92d919f41259ff791c2a09075b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:36:00.056366 5232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f0dd92d919f41259ff791c2a09075b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:36:00.062866 5237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f0dd92d919f41259ff791c2a09075b6 [term 1 LEADER]: Becoming Leader. State: Replica: 2f0dd92d919f41259ff791c2a09075b6, State: Running, Role: LEADER
I20260928 08:36:00.063902 5237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f0dd92d919f41259ff791c2a09075b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0dd92d919f41259ff791c2a09075b6" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45405 } }
I20260928 08:36:00.067703 5239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f0dd92d919f41259ff791c2a09075b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f0dd92d919f41259ff791c2a09075b6. Latest consensus state: current_term: 1 leader_uuid: "2f0dd92d919f41259ff791c2a09075b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0dd92d919f41259ff791c2a09075b6" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45405 } } }
I20260928 08:36:00.068229 5239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f0dd92d919f41259ff791c2a09075b6 [sys.catalog]: This master's current role is: LEADER
I20260928 08:36:00.069258 5251 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:36:00.068989 5238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f0dd92d919f41259ff791c2a09075b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f0dd92d919f41259ff791c2a09075b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f0dd92d919f41259ff791c2a09075b6" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45405 } } }
I20260928 08:36:00.069438 5238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f0dd92d919f41259ff791c2a09075b6 [sys.catalog]: This master's current role is: LEADER
I20260928 08:36:00.082477 5251 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:36:00.084645 5235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:00.063139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53681 (local address 127.15.134.254:45405)
0928 08:36:00.063417 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:00.063428 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:00.064501 (+ 1073us) server_negotiation.cc:407] Connection header received
0928 08:36:00.064794 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:00.064800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:00.064894 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:00.065016 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:00.067561 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:00.068487 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:00.070423 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:00.071085 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:00.071194 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:00.075794 (+ 4600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:00.075805 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:00.075813 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:00.075873 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:00.080507 (+ 4634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:00.081720 (+ 1213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:00.081728 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:00.081733 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:00.081946 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:00.082602 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:00.082607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:00.082610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:00.083033 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:00.083301 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:00.084031 (+ 730us) server_negotiation.cc:299] Negotiation successful
0928 08:36:00.084227 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260928 08:36:00.094110 5251 catalog_manager.cc:1347] Generated new cluster ID: 258a8041069d41e9a264ce200c8444d9
I20260928 08:36:00.094236 5251 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:36:00.108500 5251 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:36:00.110368 5251 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:36:00.125955 5251 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f0dd92d919f41259ff791c2a09075b6: Generated new TSK 0
I20260928 08:36:00.126844 5251 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:36:00.134949 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 5164
I20260928 08:36:00.136539 5227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:36:00.227998 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 5164
W20260928 08:36:00.246189 5149 connection.cc:535] client connection to 127.15.134.254:45405 recv error: Network error: failed to read from TLS socket (remote: 127.15.134.254:45405): Connection reset by peer (error 104)
2026-09-28T08:36:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2082 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:36:00 dist-test-slave-qp2n krb5kdc[5264](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:36:00 dist-test-slave-qp2n krb5kdc[5264](info): set up 2 sockets
Sep 28 08:36:00 dist-test-slave-qp2n krb5kdc[5264](info): commencing operation
krb5kdc: starting...
W20260928 08:36:01.342592 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:36:01 dist-test-slave-qp2n krb5kdc[5264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:36:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:36:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:36:01.618319 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:35767
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:46429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:35767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:36:01.961159 5277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:36:01.961484 5277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:36:01.961649 5277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:36:01.969137 5277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:36:01.969264 5277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:36:01.969379 5277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:36:01.969484 5277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:36:01.969571 5277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:36:01.979925 5277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:46429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:35767
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:35767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:36:01.981402 5277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:36:01.982772 5277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:36:01.995354 5282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:36:01.995391 5285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:36:01.995433 5283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:36:01.996096 5277 server_base.cc:1034] running on GCE node
I20260928 08:36:01.997066 5277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:36:01.998562 5277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:36:01.999771 5277 hybrid_clock.cc:648] HybridClock initialized: now 1790584561999734 us; error 49 us; skew 500 ppm
I20260928 08:36:02.000147 5277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:36:01 dist-test-slave-qp2n krb5kdc[5264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584561, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:36:02.003573 5277 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:36:02.005242 5277 webserver.cc:460] Webserver started at http://127.15.134.254:37159/ using document root <none> and password file <none>
I20260928 08:36:02.005959 5277 fs_manager.cc:359] Metadata directory not provided
I20260928 08:36:02.006088 5277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:36:02.006407 5277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:36:02.009572 5277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "60b976862012414d9012c858f88a2ebb"
format_stamp: "Formatted at 2026-09-28 08:36:02 on dist-test-slave-qp2n"
server_key: "54163340a729bfad55c232dc99033978"
server_key_iv: "6c026943b3297db3c756ea683e93979d"
server_key_version: "encryptionkey@0"
I20260928 08:36:02.010382 5277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "60b976862012414d9012c858f88a2ebb"
format_stamp: "Formatted at 2026-09-28 08:36:02 on dist-test-slave-qp2n"
server_key: "54163340a729bfad55c232dc99033978"
server_key_iv: "6c026943b3297db3c756ea683e93979d"
server_key_version: "encryptionkey@0"
I20260928 08:36:02.015825 5277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:36:02.019402 5292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:36:02.020771 5277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:36:02.020956 5277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "60b976862012414d9012c858f88a2ebb"
format_stamp: "Formatted at 2026-09-28 08:36:02 on dist-test-slave-qp2n"
server_key: "54163340a729bfad55c232dc99033978"
server_key_iv: "6c026943b3297db3c756ea683e93979d"
server_key_version: "encryptionkey@0"
I20260928 08:36:02.021157 5277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:36:02.037343 5277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:36:02.041623 5277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:36:02.041926 5277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:36:02.061858 5277 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:35767
I20260928 08:36:02.061894 5343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:35767 every 8 connection(s)
I20260928 08:36:02.063364 5277 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:36:02.064483 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 5277
I20260928 08:36:02.064710 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:36:02.065332 15899 external_mini_cluster.cc:1428] Setting key 7e3c196a8d0395877fe818f6b3291352
I20260928 08:36:02.068602 5344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:36:02.079339 5347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:02.067202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49141 (local address 127.15.134.254:35767)
0928 08:36:02.068182 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:02.068195 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:02.068242 (+ 47us) server_negotiation.cc:407] Connection header received
0928 08:36:02.070073 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:02.070097 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:02.070519 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:02.070988 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:02.071586 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:02.073034 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:02.074413 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:02.075124 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:02.075337 (+ 213us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:02.076767 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:02.076777 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:02.076798 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:36:02.077444 (+ 646us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:02.077535 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:36:02.077567 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:02.077792 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:02.078064 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 08:36:02.078271 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":743,"thread_start_us":465,"threads_started":1}
I20260928 08:36:02.080374 5344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60b976862012414d9012c858f88a2ebb: Bootstrap starting.
I20260928 08:36:02.083575 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:36:02.084003 5344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60b976862012414d9012c858f88a2ebb: Neither blocks nor log segments found. Creating new log.
I20260928 08:36:02.085111 5344 log.cc:824] T 00000000000000000000000000000000 P 60b976862012414d9012c858f88a2ebb: Log is configured to *not* fsync() on all Append() calls
I20260928 08:36:02.088356 5344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60b976862012414d9012c858f88a2ebb: No bootstrap required, opened a new log
I20260928 08:36:02.094769 5347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:02.088193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50635 (local address 127.15.134.254:35767)
0928 08:36:02.088354 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:02.088358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:02.088961 (+ 603us) server_negotiation.cc:407] Connection header received
0928 08:36:02.089186 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:02.089193 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:02.089264 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:02.089369 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:02.089882 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:02.090707 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:02.092071 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:02.092582 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:02.092788 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:02.093993 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:02.093998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:02.094002 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:36:02.094019 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:02.094057 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:36:02.094072 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:02.094236 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:02.094414 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 08:36:02.094468 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 08:36:02.096876 5344 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60b976862012414d9012c858f88a2ebb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b976862012414d9012c858f88a2ebb" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35767 } }
I20260928 08:36:02.097259 5344 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60b976862012414d9012c858f88a2ebb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:36:02.097381 5344 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60b976862012414d9012c858f88a2ebb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b976862012414d9012c858f88a2ebb, State: Initialized, Role: FOLLOWER
I20260928 08:36:02.098016 5344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60b976862012414d9012c858f88a2ebb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b976862012414d9012c858f88a2ebb" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35767 } }
I20260928 08:36:02.098225 5344 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60b976862012414d9012c858f88a2ebb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:36:02.098380 5344 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60b976862012414d9012c858f88a2ebb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:36:02.098577 5344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60b976862012414d9012c858f88a2ebb [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:36:02.101303 5344 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60b976862012414d9012c858f88a2ebb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b976862012414d9012c858f88a2ebb" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35767 } }
I20260928 08:36:02.101862 5344 leader_election.cc:302] T 00000000000000000000000000000000 P 60b976862012414d9012c858f88a2ebb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60b976862012414d9012c858f88a2ebb; no voters:
I20260928 08:36:02.102442 5344 leader_election.cc:288] T 00000000000000000000000000000000 P 60b976862012414d9012c858f88a2ebb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:36:02.102591 5354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60b976862012414d9012c858f88a2ebb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:36:02.104117 5354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60b976862012414d9012c858f88a2ebb [term 1 LEADER]: Becoming Leader. State: Replica: 60b976862012414d9012c858f88a2ebb, State: Running, Role: LEADER
I20260928 08:36:02.104712 5354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60b976862012414d9012c858f88a2ebb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b976862012414d9012c858f88a2ebb" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35767 } }
I20260928 08:36:02.104846 5344 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60b976862012414d9012c858f88a2ebb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:36:02.107519 5355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60b976862012414d9012c858f88a2ebb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60b976862012414d9012c858f88a2ebb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b976862012414d9012c858f88a2ebb" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35767 } } }
I20260928 08:36:02.107856 5356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60b976862012414d9012c858f88a2ebb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60b976862012414d9012c858f88a2ebb. Latest consensus state: current_term: 1 leader_uuid: "60b976862012414d9012c858f88a2ebb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60b976862012414d9012c858f88a2ebb" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35767 } } }
I20260928 08:36:02.108069 5355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60b976862012414d9012c858f88a2ebb [sys.catalog]: This master's current role is: LEADER
I20260928 08:36:02.108125 5356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60b976862012414d9012c858f88a2ebb [sys.catalog]: This master's current role is: LEADER
I20260928 08:36:02.110209 5363 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:36:02.115525 5363 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:36:02.124262 5363 catalog_manager.cc:1347] Generated new cluster ID: 61267eddb6eb4202b568faf486eb9f2f
I20260928 08:36:02.124377 5363 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:36:02.188364 5363 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:36:02.189889 5363 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:36:02.203224 5363 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60b976862012414d9012c858f88a2ebb: Generated new TSK 0
I20260928 08:36:02.204679 5363 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:36:02.226179 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 5277
W20260928 08:36:02.228034 5339 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49141
W20260928 08:36:02.228994 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 5277: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:36:02.229147 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 5277
W20260928 08:36:02.247573 5262 connection.cc:535] client connection to 127.15.134.254:35767 recv error: Network error: failed to read from TLS socket (remote: 127.15.134.254:35767): Connection reset by peer (error 104)
2026-09-28T08:36:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2001 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:36:02 dist-test-slave-qp2n krb5kdc[5375](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:36:02 dist-test-slave-qp2n krb5kdc[5375](info): set up 2 sockets
Sep 28 08:36:02 dist-test-slave-qp2n krb5kdc[5375](info): commencing operation
krb5kdc: starting...
W20260928 08:36:03.402786 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:36:03 dist-test-slave-qp2n krb5kdc[5375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:36:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:36:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:36:03.668578 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:35575
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:35263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:35575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:36:04.027607 5388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:36:04.027897 5388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:36:04.028095 5388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:36:04.035276 5388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:36:04.035384 5388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:36:04.035511 5388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:36:04.035614 5388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:36:04.035673 5388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:36:04.045719 5388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:35263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:35575
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:35575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:36:04.047079 5388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:36:04.048427 5388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:36:04.061002 5393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:36:04.060966 5397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:36:04.061105 5394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:36:04.069267 5388 server_base.cc:1034] running on GCE node
I20260928 08:36:04.070667 5388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:36:04.073546 5388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:36:04.074898 5388 hybrid_clock.cc:648] HybridClock initialized: now 1790584564074828 us; error 67 us; skew 500 ppm
I20260928 08:36:04.075456 5388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:36:04 dist-test-slave-qp2n krb5kdc[5375](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584564, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:36:04.080101 5388 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:36:04.082355 5388 webserver.cc:460] Webserver started at http://127.15.134.254:36007/ using document root <none> and password file <none>
I20260928 08:36:04.083350 5388 fs_manager.cc:359] Metadata directory not provided
I20260928 08:36:04.083582 5388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:36:04.084046 5388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:36:04.088649 5388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "9368eeb339594382a8961fec78de2029"
format_stamp: "Formatted at 2026-09-28 08:36:04 on dist-test-slave-qp2n"
server_key: "aee83c5ac1c22962c6a5eb362042dcf4"
server_key_iv: "b874ac053acf273ce48aa4cf0476e656"
server_key_version: "encryptionkey@0"
I20260928 08:36:04.089812 5388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "9368eeb339594382a8961fec78de2029"
format_stamp: "Formatted at 2026-09-28 08:36:04 on dist-test-slave-qp2n"
server_key: "aee83c5ac1c22962c6a5eb362042dcf4"
server_key_iv: "b874ac053acf273ce48aa4cf0476e656"
server_key_version: "encryptionkey@0"
I20260928 08:36:04.097548 5388 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260928 08:36:04.102387 5404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:36:04.104031 5388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260928 08:36:04.104246 5388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "9368eeb339594382a8961fec78de2029"
format_stamp: "Formatted at 2026-09-28 08:36:04 on dist-test-slave-qp2n"
server_key: "aee83c5ac1c22962c6a5eb362042dcf4"
server_key_iv: "b874ac053acf273ce48aa4cf0476e656"
server_key_version: "encryptionkey@0"
I20260928 08:36:04.104465 5388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:36:04.110535 5388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:36:04.116138 5388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:36:04.116557 5388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:36:04.135994 5388 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:35575
I20260928 08:36:04.136018 5455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:35575 every 8 connection(s)
I20260928 08:36:04.137594 5388 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:36:04.141460 5456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:36:04.144930 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 5388
I20260928 08:36:04.145112 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:36:04.145557 15899 external_mini_cluster.cc:1428] Setting key 84c21670ebe80348ec8fc11c0a68f6de
I20260928 08:36:04.153678 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:04.147419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51079 (local address 127.15.134.254:35575)
0928 08:36:04.148289 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:04.148301 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:04.148341 (+ 40us) server_negotiation.cc:407] Connection header received
0928 08:36:04.150067 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:04.150097 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:04.150516 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:04.150958 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:04.152001 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:04.152007 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:04.152020 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:36:04.152033 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:04.152120 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:36:04.152147 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:04.152379 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:04.152518 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 08:36:04.152737 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":404,"threads_started":1}
I20260928 08:36:04.153833 5456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9368eeb339594382a8961fec78de2029: Bootstrap starting.
I20260928 08:36:04.157281 5456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9368eeb339594382a8961fec78de2029: Neither blocks nor log segments found. Creating new log.
I20260928 08:36:04.157528 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:36:04.158593 5456 log.cc:824] T 00000000000000000000000000000000 P 9368eeb339594382a8961fec78de2029: Log is configured to *not* fsync() on all Append() calls
I20260928 08:36:04.162231 5456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9368eeb339594382a8961fec78de2029: No bootstrap required, opened a new log
I20260928 08:36:04.165428 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:04.162694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51333 (local address 127.15.134.254:35575)
0928 08:36:04.162901 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:04.162908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:04.163433 (+ 525us) server_negotiation.cc:407] Connection header received
0928 08:36:04.163564 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:04.163572 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:04.163634 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:04.163755 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:04.164757 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:04.164762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:04.164767 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:36:04.164785 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:04.164842 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:36:04.164863 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:04.164995 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:04.165049 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 08:36:04.165108 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:36:04.170977 5456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9368eeb339594382a8961fec78de2029 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9368eeb339594382a8961fec78de2029" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35575 } }
I20260928 08:36:04.171322 5456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9368eeb339594382a8961fec78de2029 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:36:04.171398 5456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9368eeb339594382a8961fec78de2029 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9368eeb339594382a8961fec78de2029, State: Initialized, Role: FOLLOWER
I20260928 08:36:04.172003 5456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9368eeb339594382a8961fec78de2029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9368eeb339594382a8961fec78de2029" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35575 } }
I20260928 08:36:04.172241 5456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9368eeb339594382a8961fec78de2029 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:36:04.172395 5456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9368eeb339594382a8961fec78de2029 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:36:04.172646 5456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9368eeb339594382a8961fec78de2029 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:36:04.174961 5456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9368eeb339594382a8961fec78de2029 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9368eeb339594382a8961fec78de2029" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35575 } }
I20260928 08:36:04.175469 5456 leader_election.cc:302] T 00000000000000000000000000000000 P 9368eeb339594382a8961fec78de2029 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9368eeb339594382a8961fec78de2029; no voters:
I20260928 08:36:04.176127 5456 leader_election.cc:288] T 00000000000000000000000000000000 P 9368eeb339594382a8961fec78de2029 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:36:04.176309 5466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9368eeb339594382a8961fec78de2029 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:36:04.177031 5466 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9368eeb339594382a8961fec78de2029 [term 1 LEADER]: Becoming Leader. State: Replica: 9368eeb339594382a8961fec78de2029, State: Running, Role: LEADER
I20260928 08:36:04.177618 5466 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9368eeb339594382a8961fec78de2029 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9368eeb339594382a8961fec78de2029" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35575 } }
I20260928 08:36:04.178315 5456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9368eeb339594382a8961fec78de2029 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:36:04.180500 5467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9368eeb339594382a8961fec78de2029 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9368eeb339594382a8961fec78de2029" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9368eeb339594382a8961fec78de2029" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35575 } } }
I20260928 08:36:04.180473 5468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9368eeb339594382a8961fec78de2029 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9368eeb339594382a8961fec78de2029. Latest consensus state: current_term: 1 leader_uuid: "9368eeb339594382a8961fec78de2029" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9368eeb339594382a8961fec78de2029" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 35575 } } }
I20260928 08:36:04.181047 5467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9368eeb339594382a8961fec78de2029 [sys.catalog]: This master's current role is: LEADER
I20260928 08:36:04.181047 5468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9368eeb339594382a8961fec78de2029 [sys.catalog]: This master's current role is: LEADER
I20260928 08:36:04.183097 5472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:36:04.188841 5472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:36:04.198530 5472 catalog_manager.cc:1347] Generated new cluster ID: 4a7d431fdd5d4e10980711ede0fe81b5
I20260928 08:36:04.198673 5472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:36:04.224974 5472 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:36:04.225446 5472 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:36:04.231452 5472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9368eeb339594382a8961fec78de2029: Generated new TSK 0
I20260928 08:36:04.232314 5472 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:36:04.242169 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 5388
W20260928 08:36:04.243345 5451 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51079
W20260928 08:36:04.244120 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 5388: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:36:04.244304 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 5388
2026-09-28T08:36:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2014 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:36:04 dist-test-slave-qp2n krb5kdc[5487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:36:04 dist-test-slave-qp2n krb5kdc[5487](info): set up 2 sockets
Sep 28 08:36:04 dist-test-slave-qp2n krb5kdc[5487](info): commencing operation
krb5kdc: starting...
W20260928 08:36:05.366755 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:36:05 dist-test-slave-qp2n krb5kdc[5487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:36:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:36:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:36:05.640434 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:43215
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:39543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:43215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:36:05.986691 5500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:36:05.986994 5500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:36:05.987160 5500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:36:05.994839 5500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:36:05.995018 5500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:36:05.995129 5500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:36:05.995249 5500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:36:05.995330 5500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:36:06.005679 5500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:39543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:43215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:43215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:36:06.007036 5500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:36:06.008373 5500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:36:06.020578 5505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:36:06.020691 5506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:36:06.020789 5508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:36:06.021333 5500 server_base.cc:1034] running on GCE node
I20260928 08:36:06.022297 5500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:36:06.023809 5500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:36:06.025068 5500 hybrid_clock.cc:648] HybridClock initialized: now 1790584566025021 us; error 57 us; skew 500 ppm
I20260928 08:36:06.025552 5500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:36:06 dist-test-slave-qp2n krb5kdc[5487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584566, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:36:06.029004 5500 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:36:06.030661 5500 webserver.cc:460] Webserver started at http://127.15.134.254:40849/ using document root <none> and password file <none>
I20260928 08:36:06.031394 5500 fs_manager.cc:359] Metadata directory not provided
I20260928 08:36:06.031551 5500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:36:06.031880 5500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:36:06.035145 5500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "554f3469d24d4cdfb5b2e603af298f02"
format_stamp: "Formatted at 2026-09-28 08:36:06 on dist-test-slave-qp2n"
server_key: "1f2fe6c1a0641145d3e1fb4d73859878"
server_key_iv: "9163505daf8b555d2981da51a1759f69"
server_key_version: "encryptionkey@0"
I20260928 08:36:06.036013 5500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "554f3469d24d4cdfb5b2e603af298f02"
format_stamp: "Formatted at 2026-09-28 08:36:06 on dist-test-slave-qp2n"
server_key: "1f2fe6c1a0641145d3e1fb4d73859878"
server_key_iv: "9163505daf8b555d2981da51a1759f69"
server_key_version: "encryptionkey@0"
I20260928 08:36:06.041680 5500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 08:36:06.045575 5515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:36:06.047060 5500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:36:06.047281 5500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "554f3469d24d4cdfb5b2e603af298f02"
format_stamp: "Formatted at 2026-09-28 08:36:06 on dist-test-slave-qp2n"
server_key: "1f2fe6c1a0641145d3e1fb4d73859878"
server_key_iv: "9163505daf8b555d2981da51a1759f69"
server_key_version: "encryptionkey@0"
I20260928 08:36:06.047501 5500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:36:06.061494 5500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:36:06.066954 5500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:36:06.067320 5500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:36:06.089687 5566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:43215 every 8 connection(s)
I20260928 08:36:06.089697 5500 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:43215
I20260928 08:36:06.091384 5500 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:36:06.095468 5567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:36:06.097018 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 5500
I20260928 08:36:06.097256 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:36:06.097733 15899 external_mini_cluster.cc:1428] Setting key 3505cceb8a4e3b6ff9cbd16759afb252
I20260928 08:36:06.108551 5567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 554f3469d24d4cdfb5b2e603af298f02: Bootstrap starting.
Sep 28 08:36:06 dist-test-slave-qp2n krb5kdc[5487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:36:06.111912 5567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 554f3469d24d4cdfb5b2e603af298f02: Neither blocks nor log segments found. Creating new log.
I20260928 08:36:06.113067 5567 log.cc:824] T 00000000000000000000000000000000 P 554f3469d24d4cdfb5b2e603af298f02: Log is configured to *not* fsync() on all Append() calls
I20260928 08:36:06.115990 5567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 554f3469d24d4cdfb5b2e603af298f02: No bootstrap required, opened a new log
I20260928 08:36:06.120048 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:06.099588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33667 (local address 127.15.134.254:43215)
0928 08:36:06.100400 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:06.100416 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:06.100461 (+ 45us) server_negotiation.cc:407] Connection header received
0928 08:36:06.102233 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:06.102259 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:06.102692 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:06.103115 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:06.104603 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:06.105975 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:06.107241 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:06.107914 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:06.108110 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:06.112505 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:06.112513 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:06.112529 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:06.112565 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:06.115814 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:06.116572 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:06.116579 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:06.116585 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:06.116813 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:06.117199 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:06.117203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:06.117208 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:06.117763 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:06.118133 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:06.118583 (+ 450us) server_negotiation.cc:299] Negotiation successful
0928 08:36:06.119036 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":374,"threads_started":1}
I20260928 08:36:06.124079 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:36:06.124974 5567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 554f3469d24d4cdfb5b2e603af298f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554f3469d24d4cdfb5b2e603af298f02" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 43215 } }
I20260928 08:36:06.125344 5567 raft_consensus.cc:374] T 00000000000000000000000000000000 P 554f3469d24d4cdfb5b2e603af298f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:36:06.125448 5567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 554f3469d24d4cdfb5b2e603af298f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 554f3469d24d4cdfb5b2e603af298f02, State: Initialized, Role: FOLLOWER
I20260928 08:36:06.126084 5567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 554f3469d24d4cdfb5b2e603af298f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554f3469d24d4cdfb5b2e603af298f02" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 43215 } }
I20260928 08:36:06.126343 5567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 554f3469d24d4cdfb5b2e603af298f02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:36:06.126484 5567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 554f3469d24d4cdfb5b2e603af298f02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:36:06.126657 5567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 554f3469d24d4cdfb5b2e603af298f02 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:36:06.129346 5567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 554f3469d24d4cdfb5b2e603af298f02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554f3469d24d4cdfb5b2e603af298f02" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 43215 } }
I20260928 08:36:06.130033 5567 leader_election.cc:302] T 00000000000000000000000000000000 P 554f3469d24d4cdfb5b2e603af298f02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 554f3469d24d4cdfb5b2e603af298f02; no voters:
I20260928 08:36:06.130705 5567 leader_election.cc:288] T 00000000000000000000000000000000 P 554f3469d24d4cdfb5b2e603af298f02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:36:06.130784 5577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 554f3469d24d4cdfb5b2e603af298f02 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:36:06.132092 5577 raft_consensus.cc:686] T 00000000000000000000000000000000 P 554f3469d24d4cdfb5b2e603af298f02 [term 1 LEADER]: Becoming Leader. State: Replica: 554f3469d24d4cdfb5b2e603af298f02, State: Running, Role: LEADER
I20260928 08:36:06.132819 5577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 554f3469d24d4cdfb5b2e603af298f02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554f3469d24d4cdfb5b2e603af298f02" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 43215 } }
I20260928 08:36:06.133224 5567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 554f3469d24d4cdfb5b2e603af298f02 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:36:06.136225 5579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 554f3469d24d4cdfb5b2e603af298f02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 554f3469d24d4cdfb5b2e603af298f02. Latest consensus state: current_term: 1 leader_uuid: "554f3469d24d4cdfb5b2e603af298f02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554f3469d24d4cdfb5b2e603af298f02" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 43215 } } }
I20260928 08:36:06.136631 5579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 554f3469d24d4cdfb5b2e603af298f02 [sys.catalog]: This master's current role is: LEADER
I20260928 08:36:06.139195 5578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 554f3469d24d4cdfb5b2e603af298f02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "554f3469d24d4cdfb5b2e603af298f02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "554f3469d24d4cdfb5b2e603af298f02" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 43215 } } }
I20260928 08:36:06.139701 5578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 554f3469d24d4cdfb5b2e603af298f02 [sys.catalog]: This master's current role is: LEADER
I20260928 08:36:06.140834 5584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:36:06.146354 5584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:36:06.150631 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:06.128832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38721 (local address 127.15.134.254:43215)
0928 08:36:06.129060 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:06.129070 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:06.129539 (+ 469us) server_negotiation.cc:407] Connection header received
0928 08:36:06.129718 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:06.129723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:06.129790 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:06.129872 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:06.131911 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:06.133128 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:06.135144 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:06.136155 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:06.136349 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:06.140606 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:06.140636 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:06.140643 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:06.140698 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:06.145749 (+ 5051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:06.146858 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:06.146863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:06.146866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:06.147021 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:06.147516 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:06.147522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:06.147525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:06.148095 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:06.148428 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:06.149878 (+ 1450us) server_negotiation.cc:299] Negotiation successful
0928 08:36:06.150186 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:36:06.157374 5584 catalog_manager.cc:1347] Generated new cluster ID: d2af3a5c567b4e898bcee31cea00b217
I20260928 08:36:06.157500 5584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:36:06.176151 5584 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:36:06.178117 5584 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:36:06.186137 5584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 554f3469d24d4cdfb5b2e603af298f02: Generated new TSK 0
I20260928 08:36:06.187193 5584 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:36:06.194548 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 5500
I20260928 08:36:06.195773 5562 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:36:06.287882 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 5500
2026-09-28T08:36:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2043 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:36:06 dist-test-slave-qp2n krb5kdc[5599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:36:06 dist-test-slave-qp2n krb5kdc[5599](info): set up 2 sockets
Sep 28 08:36:06 dist-test-slave-qp2n krb5kdc[5599](info): commencing operation
krb5kdc: starting...
W20260928 08:36:07.426771 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:36:07 dist-test-slave-qp2n krb5kdc[5599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:36:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:36:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:36:07.696038 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:45835
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:45835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:36:08.040784 5612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:36:08.041100 5612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:36:08.041290 5612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:36:08.048707 5612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:36:08.048822 5612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:36:08.048935 5612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:36:08.049038 5612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:36:08.049111 5612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:36:08.059237 5612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:45835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:45835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:36:08.060659 5612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:36:08.061982 5612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:36:08.073824 5617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:36:08.073900 5618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:36:08.073861 5620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:36:08.075248 5612 server_base.cc:1034] running on GCE node
I20260928 08:36:08.076076 5612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:36:08.077558 5612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:36:08.078769 5612 hybrid_clock.cc:648] HybridClock initialized: now 1790584568078728 us; error 27 us; skew 500 ppm
I20260928 08:36:08.079129 5612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:36:08 dist-test-slave-qp2n krb5kdc[5599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584568, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:36:08.082332 5612 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:36:08.083798 5612 webserver.cc:460] Webserver started at http://127.15.134.254:34293/ using document root <none> and password file <none>
I20260928 08:36:08.084497 5612 fs_manager.cc:359] Metadata directory not provided
I20260928 08:36:08.084649 5612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:36:08.084961 5612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:36:08.087891 5612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "8829baa8d0ff4a2bae7f88bbb7b5e64a"
format_stamp: "Formatted at 2026-09-28 08:36:08 on dist-test-slave-qp2n"
server_key: "7852b8c339d0b699b859211f88db0d66"
server_key_iv: "d1fda3f71324f73007037ab7b2ba2063"
server_key_version: "encryptionkey@0"
I20260928 08:36:08.088728 5612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "8829baa8d0ff4a2bae7f88bbb7b5e64a"
format_stamp: "Formatted at 2026-09-28 08:36:08 on dist-test-slave-qp2n"
server_key: "7852b8c339d0b699b859211f88db0d66"
server_key_iv: "d1fda3f71324f73007037ab7b2ba2063"
server_key_version: "encryptionkey@0"
I20260928 08:36:08.094214 5612 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 08:36:08.097649 5627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:36:08.098966 5612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:36:08.099159 5612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "8829baa8d0ff4a2bae7f88bbb7b5e64a"
format_stamp: "Formatted at 2026-09-28 08:36:08 on dist-test-slave-qp2n"
server_key: "7852b8c339d0b699b859211f88db0d66"
server_key_iv: "d1fda3f71324f73007037ab7b2ba2063"
server_key_version: "encryptionkey@0"
I20260928 08:36:08.099335 5612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:36:08.142285 5612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:36:08.147238 5612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:36:08.147567 5612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:36:08.167696 5612 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:45835
I20260928 08:36:08.167749 5680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:45835 every 8 connection(s)
I20260928 08:36:08.169207 5612 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:36:08.172899 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 5612
I20260928 08:36:08.173080 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:36:08.173105 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:36:08.173532 15899 external_mini_cluster.cc:1428] Setting key 527892e913fa9cb392730b35a2f1274c
I20260928 08:36:08.188380 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:08.175486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42797 (local address 127.15.134.254:45835)
0928 08:36:08.176332 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:08.176347 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:08.176396 (+ 49us) server_negotiation.cc:407] Connection header received
0928 08:36:08.178211 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:08.178238 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:08.178792 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:08.179241 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:08.179790 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:08.181835 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:08.183155 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:08.184167 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:08.184368 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:08.185518 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:08.185529 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:08.185549 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:36:08.186313 (+ 764us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:08.186414 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:36:08.186444 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:08.186558 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:08.186763 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 08:36:08.187033 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":350,"threads_started":1}
I20260928 08:36:08.188566 5681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8829baa8d0ff4a2bae7f88bbb7b5e64a: Bootstrap starting.
I20260928 08:36:08.191845 5681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8829baa8d0ff4a2bae7f88bbb7b5e64a: Neither blocks nor log segments found. Creating new log.
I20260928 08:36:08.192134 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:36:08.193204 5681 log.cc:824] T 00000000000000000000000000000000 P 8829baa8d0ff4a2bae7f88bbb7b5e64a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:36:08.196766 5681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8829baa8d0ff4a2bae7f88bbb7b5e64a: No bootstrap required, opened a new log
I20260928 08:36:08.203979 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:08.197477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42729 (local address 127.15.134.254:45835)
0928 08:36:08.197667 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:08.197675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:08.198074 (+ 399us) server_negotiation.cc:407] Connection header received
0928 08:36:08.198297 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:08.198302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:08.198365 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:08.198470 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:08.199089 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:08.199865 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:08.201246 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:08.201727 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:08.201886 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:08.203072 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:08.203079 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:08.203085 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:36:08.203116 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:08.203194 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:36:08.203212 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:08.203330 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:08.203458 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 08:36:08.203579 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 08:36:08.205518 5681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8829baa8d0ff4a2bae7f88bbb7b5e64a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8829baa8d0ff4a2bae7f88bbb7b5e64a" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45835 } }
I20260928 08:36:08.205943 5681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8829baa8d0ff4a2bae7f88bbb7b5e64a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:36:08.206022 5681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8829baa8d0ff4a2bae7f88bbb7b5e64a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8829baa8d0ff4a2bae7f88bbb7b5e64a, State: Initialized, Role: FOLLOWER
I20260928 08:36:08.206519 5681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8829baa8d0ff4a2bae7f88bbb7b5e64a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8829baa8d0ff4a2bae7f88bbb7b5e64a" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45835 } }
I20260928 08:36:08.206729 5681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8829baa8d0ff4a2bae7f88bbb7b5e64a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:36:08.206828 5681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8829baa8d0ff4a2bae7f88bbb7b5e64a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:36:08.206947 5681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8829baa8d0ff4a2bae7f88bbb7b5e64a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:36:08.209084 5681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8829baa8d0ff4a2bae7f88bbb7b5e64a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8829baa8d0ff4a2bae7f88bbb7b5e64a" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45835 } }
I20260928 08:36:08.209522 5681 leader_election.cc:302] T 00000000000000000000000000000000 P 8829baa8d0ff4a2bae7f88bbb7b5e64a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8829baa8d0ff4a2bae7f88bbb7b5e64a; no voters:
I20260928 08:36:08.210067 5681 leader_election.cc:288] T 00000000000000000000000000000000 P 8829baa8d0ff4a2bae7f88bbb7b5e64a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:36:08.210218 5691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8829baa8d0ff4a2bae7f88bbb7b5e64a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:36:08.210956 5691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8829baa8d0ff4a2bae7f88bbb7b5e64a [term 1 LEADER]: Becoming Leader. State: Replica: 8829baa8d0ff4a2bae7f88bbb7b5e64a, State: Running, Role: LEADER
I20260928 08:36:08.211612 5691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8829baa8d0ff4a2bae7f88bbb7b5e64a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8829baa8d0ff4a2bae7f88bbb7b5e64a" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45835 } }
I20260928 08:36:08.212440 5681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8829baa8d0ff4a2bae7f88bbb7b5e64a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:36:08.214530 5693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8829baa8d0ff4a2bae7f88bbb7b5e64a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8829baa8d0ff4a2bae7f88bbb7b5e64a. Latest consensus state: current_term: 1 leader_uuid: "8829baa8d0ff4a2bae7f88bbb7b5e64a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8829baa8d0ff4a2bae7f88bbb7b5e64a" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45835 } } }
I20260928 08:36:08.214983 5693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8829baa8d0ff4a2bae7f88bbb7b5e64a [sys.catalog]: This master's current role is: LEADER
I20260928 08:36:08.215124 5692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8829baa8d0ff4a2bae7f88bbb7b5e64a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8829baa8d0ff4a2bae7f88bbb7b5e64a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8829baa8d0ff4a2bae7f88bbb7b5e64a" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45835 } } }
I20260928 08:36:08.215459 5692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8829baa8d0ff4a2bae7f88bbb7b5e64a [sys.catalog]: This master's current role is: LEADER
I20260928 08:36:08.217916 5699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:36:08.223644 5699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:36:08.233695 5699 catalog_manager.cc:1347] Generated new cluster ID: e4101c6227114abe81dbc94c1c84e410
I20260928 08:36:08.233847 5699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:36:08.257550 5699 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:36:08.259037 5699 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:36:08.266999 5699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8829baa8d0ff4a2bae7f88bbb7b5e64a: Generated new TSK 0
I20260928 08:36:08.268149 5699 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:36:08.283006 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 5612
W20260928 08:36:08.284849 5676 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42797
W20260928 08:36:08.285606 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 5612: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:36:08.285775 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 5612
2026-09-28T08:36:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1998 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:36:08 dist-test-slave-qp2n krb5kdc[5712](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:36:08 dist-test-slave-qp2n krb5kdc[5712](info): set up 2 sockets
Sep 28 08:36:08 dist-test-slave-qp2n krb5kdc[5712](info): commencing operation
krb5kdc: starting...
W20260928 08:36:09.410997 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:36:09 dist-test-slave-qp2n krb5kdc[5712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:36:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:36:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:36:09.681082 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:37795
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:37391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:37795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:36:10.024358 5726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:36:10.024739 5726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:36:10.024914 5726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:36:10.032407 5726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:36:10.032539 5726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:36:10.032691 5726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:36:10.032799 5726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:36:10.032874 5726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:36:10.043258 5726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:37391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:37795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:37795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:36:10.044775 5726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:36:10.046172 5726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:36:10.058911 5731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:36:10.058923 5732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:36:10.059345 5734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:36:10.059608 5726 server_base.cc:1034] running on GCE node
I20260928 08:36:10.060529 5726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:36:10.062197 5726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:36:10.063436 5726 hybrid_clock.cc:648] HybridClock initialized: now 1790584570063375 us; error 59 us; skew 500 ppm
I20260928 08:36:10.063805 5726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:36:10 dist-test-slave-qp2n krb5kdc[5712](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584570, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:36:10.073884 5726 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:36:10.075435 5726 webserver.cc:460] Webserver started at http://127.15.134.254:45841/ using document root <none> and password file <none>
I20260928 08:36:10.076159 5726 fs_manager.cc:359] Metadata directory not provided
I20260928 08:36:10.076263 5726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:36:10.076557 5726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:36:10.079730 5726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "99c6dad3127e42e7beddb7a773abd19e"
format_stamp: "Formatted at 2026-09-28 08:36:10 on dist-test-slave-qp2n"
server_key: "0e313ca116e3ee2a5a6eab48f7de4974"
server_key_iv: "745e437aa407add690e5b2ec5c4adbeb"
server_key_version: "encryptionkey@0"
I20260928 08:36:10.080479 5726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "99c6dad3127e42e7beddb7a773abd19e"
format_stamp: "Formatted at 2026-09-28 08:36:10 on dist-test-slave-qp2n"
server_key: "0e313ca116e3ee2a5a6eab48f7de4974"
server_key_iv: "745e437aa407add690e5b2ec5c4adbeb"
server_key_version: "encryptionkey@0"
I20260928 08:36:10.086223 5726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:36:10.089824 5741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:36:10.091293 5726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:36:10.091454 5726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "99c6dad3127e42e7beddb7a773abd19e"
format_stamp: "Formatted at 2026-09-28 08:36:10 on dist-test-slave-qp2n"
server_key: "0e313ca116e3ee2a5a6eab48f7de4974"
server_key_iv: "745e437aa407add690e5b2ec5c4adbeb"
server_key_version: "encryptionkey@0"
I20260928 08:36:10.091639 5726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:36:10.097550 5726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:36:10.101815 5726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:36:10.102070 5726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:36:10.120798 5726 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:37795
I20260928 08:36:10.120843 5792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:37795 every 8 connection(s)
I20260928 08:36:10.122268 5726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:36:10.126291 5793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:36:10.127529 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 5726
I20260928 08:36:10.127731 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:36:10.128170 15899 external_mini_cluster.cc:1428] Setting key 241b168b3cc9c40070448162ddf4635e
I20260928 08:36:10.137056 5796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:10.130089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42103 (local address 127.15.134.254:37795)
0928 08:36:10.130938 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:10.130954 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:10.131012 (+ 58us) server_negotiation.cc:407] Connection header received
0928 08:36:10.133074 (+ 2062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:10.133105 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:10.133609 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:10.134092 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:10.135066 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:10.135074 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:10.135092 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:36:10.135108 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:10.135174 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:36:10.135208 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:10.135420 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:10.135544 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 08:36:10.135759 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":347,"threads_started":1}
I20260928 08:36:10.141291 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:36:10.141659 5793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99c6dad3127e42e7beddb7a773abd19e: Bootstrap starting.
I20260928 08:36:10.145069 5793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99c6dad3127e42e7beddb7a773abd19e: Neither blocks nor log segments found. Creating new log.
I20260928 08:36:10.146533 5793 log.cc:824] T 00000000000000000000000000000000 P 99c6dad3127e42e7beddb7a773abd19e: Log is configured to *not* fsync() on all Append() calls
I20260928 08:36:10.148398 5796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:10.145869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37897 (local address 127.15.134.254:37795)
0928 08:36:10.146062 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:10.146070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:10.146436 (+ 366us) server_negotiation.cc:407] Connection header received
0928 08:36:10.146632 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:10.146642 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:10.146721 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:10.146827 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:10.147755 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:10.147761 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:10.147765 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:36:10.147783 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:10.147829 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:36:10.147844 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:10.147957 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:10.148069 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 08:36:10.148129 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 08:36:10.149825 5793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99c6dad3127e42e7beddb7a773abd19e: No bootstrap required, opened a new log
I20260928 08:36:10.157574 5793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99c6dad3127e42e7beddb7a773abd19e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c6dad3127e42e7beddb7a773abd19e" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 37795 } }
I20260928 08:36:10.157930 5793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99c6dad3127e42e7beddb7a773abd19e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:36:10.158004 5793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99c6dad3127e42e7beddb7a773abd19e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99c6dad3127e42e7beddb7a773abd19e, State: Initialized, Role: FOLLOWER
I20260928 08:36:10.158551 5793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99c6dad3127e42e7beddb7a773abd19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c6dad3127e42e7beddb7a773abd19e" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 37795 } }
I20260928 08:36:10.158754 5793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99c6dad3127e42e7beddb7a773abd19e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:36:10.158852 5793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99c6dad3127e42e7beddb7a773abd19e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:36:10.158973 5793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99c6dad3127e42e7beddb7a773abd19e [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:36:10.161500 5793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99c6dad3127e42e7beddb7a773abd19e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c6dad3127e42e7beddb7a773abd19e" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 37795 } }
I20260928 08:36:10.161993 5793 leader_election.cc:302] T 00000000000000000000000000000000 P 99c6dad3127e42e7beddb7a773abd19e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99c6dad3127e42e7beddb7a773abd19e; no voters:
I20260928 08:36:10.162588 5793 leader_election.cc:288] T 00000000000000000000000000000000 P 99c6dad3127e42e7beddb7a773abd19e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:36:10.162734 5803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99c6dad3127e42e7beddb7a773abd19e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:36:10.163376 5803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99c6dad3127e42e7beddb7a773abd19e [term 1 LEADER]: Becoming Leader. State: Replica: 99c6dad3127e42e7beddb7a773abd19e, State: Running, Role: LEADER
I20260928 08:36:10.163962 5803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99c6dad3127e42e7beddb7a773abd19e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c6dad3127e42e7beddb7a773abd19e" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 37795 } }
I20260928 08:36:10.164772 5793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99c6dad3127e42e7beddb7a773abd19e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:36:10.166591 5804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99c6dad3127e42e7beddb7a773abd19e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99c6dad3127e42e7beddb7a773abd19e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c6dad3127e42e7beddb7a773abd19e" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 37795 } } }
I20260928 08:36:10.166942 5804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99c6dad3127e42e7beddb7a773abd19e [sys.catalog]: This master's current role is: LEADER
I20260928 08:36:10.167098 5805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99c6dad3127e42e7beddb7a773abd19e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99c6dad3127e42e7beddb7a773abd19e. Latest consensus state: current_term: 1 leader_uuid: "99c6dad3127e42e7beddb7a773abd19e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99c6dad3127e42e7beddb7a773abd19e" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 37795 } } }
I20260928 08:36:10.167443 5805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99c6dad3127e42e7beddb7a773abd19e [sys.catalog]: This master's current role is: LEADER
I20260928 08:36:10.169378 5809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:36:10.175068 5809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:36:10.185153 5809 catalog_manager.cc:1347] Generated new cluster ID: f9458dddc076480392d20151eda71a6a
I20260928 08:36:10.185292 5809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:36:10.241528 5809 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:36:10.242053 5809 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:36:10.261422 5809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99c6dad3127e42e7beddb7a773abd19e: Generated new TSK 0
I20260928 08:36:10.262394 5809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:36:10.277165 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 5726
W20260928 08:36:10.278388 5788 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42103
W20260928 08:36:10.279115 15899 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 5726: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:36:10.279253 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 5726
2026-09-28T08:36:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1993 ms)
[----------] 18 tests from AuthTokenIssuingTest (36272 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:36:10 dist-test-slave-qp2n krb5kdc[5824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:36:10 dist-test-slave-qp2n krb5kdc[5824](info): set up 2 sockets
Sep 28 08:36:10 dist-test-slave-qp2n krb5kdc[5824](info): commencing operation
krb5kdc: starting...
W20260928 08:36:11.398847 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:36:11 dist-test-slave-qp2n krb5kdc[5824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:36:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:36:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:36:11.676136 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:45883
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:35099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:45883,127.15.134.253:45467,127.15.134.252:40233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:36:12.014552 5837 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 08:36:12.014915 5837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:36:12.015033 5837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:36:12.015163 5837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:36:12.022339 5837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:36:12.022461 5837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:36:12.022583 5837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:36:12.022667 5837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:36:12.022771 5837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:36:12.032572 5837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:35099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:45883,127.15.134.253:45467,127.15.134.252:40233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:45883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:36:12.034015 5837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:36:12.035332 5837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:36:12.047401 5842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:36:12.047452 5845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:36:12.047425 5843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:36:12.048812 5837 server_base.cc:1034] running on GCE node
I20260928 08:36:12.049685 5837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:36:12.051249 5837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:36:12.052474 5837 hybrid_clock.cc:648] HybridClock initialized: now 1790584572052398 us; error 49 us; skew 500 ppm
I20260928 08:36:12.052873 5837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:36:12 dist-test-slave-qp2n krb5kdc[5824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584572, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:36:12.056303 5837 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:36:12.057916 5837 webserver.cc:460] Webserver started at http://127.15.134.254:42449/ using document root <none> and password file <none>
I20260928 08:36:12.058612 5837 fs_manager.cc:359] Metadata directory not provided
I20260928 08:36:12.058709 5837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:36:12.058979 5837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:36:12.062107 5837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "97182f293fae4db8bbe96f60774a8df0"
format_stamp: "Formatted at 2026-09-28 08:36:12 on dist-test-slave-qp2n"
server_key: "ac86b2a69ea51bd6fe06676d6920dbe4"
server_key_iv: "40d48e9d04e27415d0838618325ff130"
server_key_version: "encryptionkey@0"
I20260928 08:36:12.062896 5837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "97182f293fae4db8bbe96f60774a8df0"
format_stamp: "Formatted at 2026-09-28 08:36:12 on dist-test-slave-qp2n"
server_key: "ac86b2a69ea51bd6fe06676d6920dbe4"
server_key_iv: "40d48e9d04e27415d0838618325ff130"
server_key_version: "encryptionkey@0"
I20260928 08:36:12.068156 5837 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:36:12.071861 5853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:36:12.073385 5837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:36:12.073589 5837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "97182f293fae4db8bbe96f60774a8df0"
format_stamp: "Formatted at 2026-09-28 08:36:12 on dist-test-slave-qp2n"
server_key: "ac86b2a69ea51bd6fe06676d6920dbe4"
server_key_iv: "40d48e9d04e27415d0838618325ff130"
server_key_version: "encryptionkey@0"
I20260928 08:36:12.073836 5837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:36:12.097484 5837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:36:12.102139 5837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:36:12.102427 5837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:36:12.121556 5837 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:45883
I20260928 08:36:12.121570 5905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:45883 every 8 connection(s)
I20260928 08:36:12.122943 5837 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:36:12.126688 5906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:36:12.132524 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 5837
I20260928 08:36:12.132889 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:36:12.133540 15899 external_mini_cluster.cc:1428] Setting key 86ac988cb48f31fcd42c4d47430af1ce
I20260928 08:36:12.133217 5906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45883 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:36:12 dist-test-slave-qp2n krb5kdc[5824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584572, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:36:12.158886 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.136501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:45883 (local address 127.0.0.1:49898)
0928 08:36:12.137444 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.137505 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 08:36:12.140427 (+ 2922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:36:12.141275 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:36:12.141286 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:36:12.142094 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:36:12.142591 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:12.142618 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:12.144148 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:12.144151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:12.145347 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:12.145359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:12.146759 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:12.146765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:12.146891 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:12.147660 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:36:12.147685 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:36:12.150222 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:36:12.154813 (+ 4591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:12.154824 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:12.154833 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:12.155238 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:12.155706 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:12.155710 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:12.155712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:12.155857 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:12.156885 (+ 1028us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:36:12.156891 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:36:12.157090 (+ 199us) client_negotiation.cc:769] Sending connection context
0928 08:36:12.157398 (+ 308us) client_negotiation.cc:240] Negotiation successful
0928 08:36:12.157786 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":386,"threads_started":1}
I20260928 08:36:12.158927 5909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.136501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49898 (local address 127.15.134.254:45883)
0928 08:36:12.138532 (+ 2031us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:12.138538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:12.138582 (+ 44us) server_negotiation.cc:407] Connection header received
0928 08:36:12.140805 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:12.140811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:12.141029 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:12.141195 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:12.142822 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:12.144008 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:12.145501 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:12.146513 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:12.146725 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:12.150468 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:12.150481 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:12.150500 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:12.150543 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:12.154674 (+ 4131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:12.155388 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:12.155400 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:12.155407 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:12.155596 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:12.155979 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:12.155987 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:12.155993 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:12.156752 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:12.156967 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:12.157591 (+ 624us) server_negotiation.cc:299] Negotiation successful
0928 08:36:12.158000 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":1854,"spinlock_wait_cycles":25984,"thread_start_us":1550,"threads_started":1}
WARNING: no policy specified for kudu/127.15.134.253@KRBTEST.COM; defaulting to no policy
I20260928 08:36:12.162134 5906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:36:12.163321 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.162914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:45467 (local address 127.0.0.1:54482)
0928 08:36:12.163109 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.163217 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 08:36:12.163609 5857 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.134.253:45467: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.253@KRBTEST.COM" created.
I20260928 08:36:12.165485 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.164983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:45467 (local address 127.0.0.1:54494)
0928 08:36:12.165170 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.165304 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 08:36:12.165802 5906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.253:45467: Network error: Client connection negotiation failed: client connection to 127.15.134.253:45467: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.253.keytab.
Entry for principal kudu/127.15.134.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.253.keytab.
I20260928 08:36:12.196138 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.253:45467
--webserver_interface=127.15.134.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.253.keytab
--principal=kudu/127.15.134.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:35099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:45883,127.15.134.253:45467,127.15.134.252:40233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:36:12.215194 5906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 } attempt: 1
I20260928 08:36:12.216414 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.215945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:45467 (local address 127.0.0.1:54500)
0928 08:36:12.216150 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.216271 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 08:36:12.217646 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.217265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:45467 (local address 127.0.0.1:54514)
0928 08:36:12.217439 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.217514 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 08:36:12.217861 5906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.253:45467: Network error: Client connection negotiation failed: client connection to 127.15.134.253:45467: connect: Connection refused (error 111)
I20260928 08:36:12.286106 5906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 } attempt: 2
I20260928 08:36:12.287276 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.286758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:45467 (local address 127.0.0.1:54526)
0928 08:36:12.287027 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.287130 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 08:36:12.288301 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.287948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:45467 (local address 127.0.0.1:54536)
0928 08:36:12.288096 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.288172 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 08:36:12.288484 5906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.253:45467: Network error: Client connection negotiation failed: client connection to 127.15.134.253:45467: connect: Connection refused (error 111)
I20260928 08:36:12.379725 5906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 } attempt: 3
I20260928 08:36:12.380949 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.380364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:45467 (local address 127.0.0.1:54544)
0928 08:36:12.380720 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.380816 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 08:36:12.382030 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.381646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:45467 (local address 127.0.0.1:54560)
0928 08:36:12.381825 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.381896 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260928 08:36:12.382225 5906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.253:45467: Network error: Client connection negotiation failed: client connection to 127.15.134.253:45467: connect: Connection refused (error 111)
I20260928 08:36:12.525460 5906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 } attempt: 4
I20260928 08:36:12.526649 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.526103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:45467 (local address 127.0.0.1:54568)
0928 08:36:12.526418 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.526512 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260928 08:36:12.527601 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.527269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:45467 (local address 127.0.0.1:54584)
0928 08:36:12.527406 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.527490 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 08:36:12.527808 5906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.253:45467: Network error: Client connection negotiation failed: client connection to 127.15.134.253:45467: connect: Connection refused (error 111)
W20260928 08:36:12.538955 5913 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 08:36:12.539373 5913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:36:12.539520 5913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:36:12.539670 5913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:36:12.547442 5913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:36:12.547605 5913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:36:12.547724 5913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:36:12.547821 5913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:36:12.547921 5913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:36:12.558229 5913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:35099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:45883,127.15.134.253:45467,127.15.134.252:40233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.253.keytab
--principal=kudu/127.15.134.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.253:45467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.134.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:36:12.559659 5913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:36:12.561143 5913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:36:12.575781 5921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:36:12.575754 5918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:36:12.576236 5919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:36:12.576710 5913 server_base.cc:1034] running on GCE node
I20260928 08:36:12.577593 5913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:36:12.579196 5913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:36:12.580458 5913 hybrid_clock.cc:648] HybridClock initialized: now 1790584572580388 us; error 38 us; skew 500 ppm
I20260928 08:36:12.580940 5913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:36:12 dist-test-slave-qp2n krb5kdc[5824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584572, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:36:12.584342 5913 init.cc:377] Logged in from keytab as kudu/127.15.134.253@KRBTEST.COM (short username kudu)
I20260928 08:36:12.586059 5913 webserver.cc:460] Webserver started at http://127.15.134.253:33889/ using document root <none> and password file <none>
I20260928 08:36:12.586822 5913 fs_manager.cc:359] Metadata directory not provided
I20260928 08:36:12.586947 5913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:36:12.587298 5913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:36:12.590457 5913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-1/data/instance:
uuid: "fcf02e5e6981450a912dcf5a764410b6"
format_stamp: "Formatted at 2026-09-28 08:36:12 on dist-test-slave-qp2n"
server_key: "433d8694992d6243d0ca153e7b153464"
server_key_iv: "baa8e2b2791cbb2adc9f8e993a45b27f"
server_key_version: "encryptionkey@0"
I20260928 08:36:12.591370 5913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-1/wal/instance:
uuid: "fcf02e5e6981450a912dcf5a764410b6"
format_stamp: "Formatted at 2026-09-28 08:36:12 on dist-test-slave-qp2n"
server_key: "433d8694992d6243d0ca153e7b153464"
server_key_iv: "baa8e2b2791cbb2adc9f8e993a45b27f"
server_key_version: "encryptionkey@0"
I20260928 08:36:12.597048 5913 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260928 08:36:12.600939 5928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:36:12.602463 5913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:36:12.602659 5913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-1/wal
uuid: "fcf02e5e6981450a912dcf5a764410b6"
format_stamp: "Formatted at 2026-09-28 08:36:12 on dist-test-slave-qp2n"
server_key: "433d8694992d6243d0ca153e7b153464"
server_key_iv: "baa8e2b2791cbb2adc9f8e993a45b27f"
server_key_version: "encryptionkey@0"
I20260928 08:36:12.602916 5913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:36:12.617615 5913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:36:12.622120 5913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:36:12.622398 5913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:36:12.641114 5913 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.253:45467
I20260928 08:36:12.641137 5980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.253:45467 every 8 connection(s)
I20260928 08:36:12.642534 5913 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-1/data/info.pb
I20260928 08:36:12.646281 5981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:36:12.652581 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 5913
I20260928 08:36:12.652814 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-1/wal/instance
I20260928 08:36:12.653347 15899 external_mini_cluster.cc:1428] Setting key 6917acbeb3074869fae03f14513f1e4e
I20260928 08:36:12.653250 5981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45883 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:36:12 dist-test-slave-qp2n krb5kdc[5824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584572, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.253@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:36:12.677917 5909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.656644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49910 (local address 127.15.134.254:45883)
0928 08:36:12.656871 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:12.656877 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:12.658072 (+ 1195us) server_negotiation.cc:407] Connection header received
0928 08:36:12.661029 (+ 2957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:12.661033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:12.661118 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:12.661227 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:12.663345 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:12.664255 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:12.665843 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:12.666648 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:12.666805 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:12.670364 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:12.670374 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:12.670382 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:12.670433 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:12.674273 (+ 3840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:12.675073 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:12.675082 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:12.675088 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:12.675246 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:12.675784 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:12.675792 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:12.675798 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:12.676203 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:12.676371 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:12.677166 (+ 795us) server_negotiation.cc:299] Negotiation successful
0928 08:36:12.677450 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 08:36:12.679143 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.657038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:45883 (local address 127.0.0.1:49910)
0928 08:36:12.657878 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.657927 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 08:36:12.660709 (+ 2782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:36:12.661510 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:36:12.661525 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:36:12.662336 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:36:12.663163 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:12.663190 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:12.664424 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:12.664428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:12.665699 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:12.665712 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:12.666869 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:12.666874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:12.667064 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:12.667781 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:36:12.667804 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:36:12.670103 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:36:12.674443 (+ 4340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:12.674451 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:12.674466 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:12.674932 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:12.675397 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:12.675402 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:12.675405 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:12.675664 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:12.676334 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:36:12.676340 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:36:12.676758 (+ 418us) client_negotiation.cc:769] Sending connection context
0928 08:36:12.677011 (+ 253us) client_negotiation.cc:240] Negotiation successful
0928 08:36:12.677669 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":317,"threads_started":1}
I20260928 08:36:12.681187 5981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.15.134.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.252@KRBTEST.COM" created.
Sep 28 08:36:12 dist-test-slave-qp2n krb5kdc[5824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584572, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.253@KRBTEST.COM for kudu/127.15.134.253@KRBTEST.COM
I20260928 08:36:12.697623 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.682102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:45467 (local address 127.0.0.1:54590)
0928 08:36:12.682316 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.682364 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 08:36:12.682507 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:36:12.683262 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:36:12.683266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:36:12.683616 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:36:12.683835 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:12.683844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:12.685349 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:12.685357 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:12.686687 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:12.686706 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:12.687550 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:12.687559 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:12.687777 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:12.688446 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:36:12.688473 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:36:12.691131 (+ 2658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:36:12.694890 (+ 3759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:12.694899 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:12.694906 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:12.695319 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:12.695728 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:12.695735 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:12.695739 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:12.695891 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:12.696516 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:36:12.696523 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:36:12.696699 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 08:36:12.696856 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 08:36:12.697188 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 08:36:12.697758 5986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.682133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54590 (local address 127.15.134.253:45467)
0928 08:36:12.682843 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:12.682850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:12.682878 (+ 28us) server_negotiation.cc:407] Connection header received
0928 08:36:12.682923 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:12.682929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:12.683113 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:12.683236 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:12.684014 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:12.685201 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:12.686833 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:12.687367 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:12.687462 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:12.691368 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:12.691381 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:12.691397 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:12.691428 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:12.694741 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:12.695460 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:12.695474 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:12.695477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:12.695601 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:12.696001 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:12.696009 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:12.696013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:12.696381 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:12.696658 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:12.697045 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 08:36:12.697327 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":479,"thread_start_us":292,"threads_started":1}
I20260928 08:36:12.700273 5981 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:36:12.701630 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.701173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:40233 (local address 127.0.0.1:48254)
0928 08:36:12.701369 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.701486 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260928 08:36:12.701996 5932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
I20260928 08:36:12.703589 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.703273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:40233 (local address 127.0.0.1:48266)
0928 08:36:12.703415 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.703494 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 08:36:12.703819 5981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:40233: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.252.keytab.
Entry for principal kudu/127.15.134.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.252.keytab.
I20260928 08:36:12.719666 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.252:40233
--webserver_interface=127.15.134.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.252.keytab
--principal=kudu/127.15.134.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:35099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:45883,127.15.134.253:45467,127.15.134.252:40233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:36:12.753140 5981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } attempt: 1
I20260928 08:36:12.754513 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.753948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:40233 (local address 127.0.0.1:48268)
0928 08:36:12.754182 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.754318 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 08:36:12.755666 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.755268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:40233 (local address 127.0.0.1:48272)
0928 08:36:12.755441 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.755535 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 08:36:12.755908 5981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:40233: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
I20260928 08:36:12.824159 5981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } attempt: 2
I20260928 08:36:12.825551 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.824946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:40233 (local address 127.0.0.1:48280)
0928 08:36:12.825207 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.825341 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 08:36:12.826553 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.826184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:40233 (local address 127.0.0.1:48296)
0928 08:36:12.826330 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.826412 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 08:36:12.826823 5981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:40233: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
I20260928 08:36:12.827033 5906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 } attempt: 5
Sep 28 08:36:12 dist-test-slave-qp2n krb5kdc[5824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584572, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for kudu/127.15.134.253@KRBTEST.COM
I20260928 08:36:12.840982 5986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.827753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54604 (local address 127.15.134.253:45467)
0928 08:36:12.827919 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:12.827925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:12.827948 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:36:12.828131 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:12.828136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:12.828231 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:12.828334 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:12.829163 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:12.830246 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:12.831397 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:12.832083 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:12.832203 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:12.835646 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:12.835651 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:12.835655 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:12.835685 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:12.838310 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:12.839120 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:12.839124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:12.839126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:12.839278 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:12.839673 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:12.839677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:12.839679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:12.839959 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:12.840073 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:12.840504 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 08:36:12.840754 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 08:36:12.841100 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.827681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:45467 (local address 127.0.0.1:54604)
0928 08:36:12.827884 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.827899 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:36:12.828000 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:36:12.828384 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:36:12.828387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:36:12.828841 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:36:12.829022 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:12.829031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:12.830402 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:12.830407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:12.831244 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:12.831263 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:12.832245 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:12.832249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:12.832412 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:12.833035 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:36:12.833054 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:36:12.835462 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:36:12.838460 (+ 2998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:12.838468 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:12.838474 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:12.839004 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:12.839388 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:12.839393 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:12.839397 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:12.839563 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:12.840075 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:36:12.840079 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:36:12.840254 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 08:36:12.840413 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 08:36:12.840753 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 08:36:12.842159 5906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:36:12.843330 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.842848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:40233 (local address 127.0.0.1:48310)
0928 08:36:12.843049 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.843141 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 08:36:12.844439 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.843979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:40233 (local address 127.0.0.1:48314)
0928 08:36:12.844157 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.844280 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 08:36:12.844684 5906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:40233: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
I20260928 08:36:12.895952 5906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } attempt: 1
I20260928 08:36:12.897320 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.896762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:40233 (local address 127.0.0.1:48316)
0928 08:36:12.896998 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.897110 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 08:36:12.898540 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.898158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:40233 (local address 127.0.0.1:48328)
0928 08:36:12.898313 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.898403 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 08:36:12.898836 5906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:40233: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
I20260928 08:36:12.918057 5981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } attempt: 3
I20260928 08:36:12.919278 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.918741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:40233 (local address 127.0.0.1:48334)
0928 08:36:12.918957 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.919077 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 08:36:12.920372 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.919893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:40233 (local address 127.0.0.1:48350)
0928 08:36:12.920130 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.920235 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260928 08:36:12.920701 5981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:40233: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
I20260928 08:36:12.967096 5906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } attempt: 2
I20260928 08:36:12.968282 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.967798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:40233 (local address 127.0.0.1:48366)
0928 08:36:12.968020 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.968110 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 08:36:12.969360 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:12.969004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:40233 (local address 127.0.0.1:48380)
0928 08:36:12.969180 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:12.969258 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260928 08:36:12.969571 5906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:40233: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
I20260928 08:36:13.063920 5981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } attempt: 4
I20260928 08:36:13.065225 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:13.064598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:40233 (local address 127.0.0.1:48392)
0928 08:36:13.064915 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:13.065020 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260928 08:36:13.066274 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:13.065895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:40233 (local address 127.0.0.1:48400)
0928 08:36:13.066050 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:13.066139 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 08:36:13.066481 5981 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:40233: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
W20260928 08:36:13.069201 5989 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 08:36:13.069514 5989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:36:13.069582 5989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:36:13.069662 5989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 08:36:13.075771 5906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } attempt: 3
I20260928 08:36:13.076977 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:13.076463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:40233 (local address 127.0.0.1:48402)
0928 08:36:13.076737 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:13.076823 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 08:36:13.077392 5989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:36:13.077499 5989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:36:13.077669 5989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:36:13.077741 5989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:36:13.077847 5989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:36:13.078195 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:13.077781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:40233 (local address 127.0.0.1:48410)
0928 08:36:13.078002 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:13.078079 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260928 08:36:13.078440 5906 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:40233: Network error: Client connection negotiation failed: client connection to 127.15.134.252:40233: connect: Connection refused (error 111)
I20260928 08:36:13.088124 5989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:35099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:45883,127.15.134.253:45467,127.15.134.252:40233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.252.keytab
--principal=kudu/127.15.134.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.252:40233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.134.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:36:13.089573 5989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:36:13.090924 5989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:36:13.102890 5996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:36:13.102963 5998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:36:13.102936 5995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:36:13.103516 5989 server_base.cc:1034] running on GCE node
I20260928 08:36:13.104291 5989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:36:13.105800 5989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:36:13.107121 5989 hybrid_clock.cc:648] HybridClock initialized: now 1790584573107037 us; error 41 us; skew 500 ppm
I20260928 08:36:13.107668 5989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:36:13 dist-test-slave-qp2n krb5kdc[5824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584573, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:36:13.110929 5989 init.cc:377] Logged in from keytab as kudu/127.15.134.252@KRBTEST.COM (short username kudu)
I20260928 08:36:13.112296 5989 webserver.cc:460] Webserver started at http://127.15.134.252:43129/ using document root <none> and password file <none>
I20260928 08:36:13.113035 5989 fs_manager.cc:359] Metadata directory not provided
I20260928 08:36:13.113157 5989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:36:13.113472 5989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:36:13.116920 5989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-2/data/instance:
uuid: "39df30f6d0db40b48db404b394a9cdd9"
format_stamp: "Formatted at 2026-09-28 08:36:13 on dist-test-slave-qp2n"
server_key: "bafe294986806995515035371efde93c"
server_key_iv: "6bd5e55568565c254e8179f4bdbd7c3a"
server_key_version: "encryptionkey@0"
I20260928 08:36:13.117830 5989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-2/wal/instance:
uuid: "39df30f6d0db40b48db404b394a9cdd9"
format_stamp: "Formatted at 2026-09-28 08:36:13 on dist-test-slave-qp2n"
server_key: "bafe294986806995515035371efde93c"
server_key_iv: "6bd5e55568565c254e8179f4bdbd7c3a"
server_key_version: "encryptionkey@0"
I20260928 08:36:13.123407 5989 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 08:36:13.127542 6005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:36:13.129066 5989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 08:36:13.129267 5989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-2/wal
uuid: "39df30f6d0db40b48db404b394a9cdd9"
format_stamp: "Formatted at 2026-09-28 08:36:13 on dist-test-slave-qp2n"
server_key: "bafe294986806995515035371efde93c"
server_key_iv: "6bd5e55568565c254e8179f4bdbd7c3a"
server_key_version: "encryptionkey@0"
I20260928 08:36:13.129523 5989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:36:13.145454 5989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:36:13.150241 5989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:36:13.150540 5989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:36:13.169250 5989 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.252:40233
I20260928 08:36:13.169271 6056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.252:40233 every 8 connection(s)
I20260928 08:36:13.170662 5989 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-2/data/info.pb
I20260928 08:36:13.174378 6057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:36:13.176632 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 5989
I20260928 08:36:13.176826 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790584205252592-15899-0/minicluster-data/master-2/wal/instance
I20260928 08:36:13.177366 15899 external_mini_cluster.cc:1428] Setting key 90d40363acaa43bf7b7a1f1d34d7c316
I20260928 08:36:13.182996 6057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45883 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:36:13 dist-test-slave-qp2n krb5kdc[5824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.252@KRBTEST.COM
Sep 28 08:36:13 dist-test-slave-qp2n krb5kdc[5824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584573, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.252@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:36:13.202613 6060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:13.179307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48424 (local address 127.15.134.252:40233)
0928 08:36:13.180291 (+ 984us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:13.180310 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:13.180683 (+ 373us) server_negotiation.cc:407] Connection header received
0928 08:36:13.182506 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:13.182529 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:13.182956 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:13.183312 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:13.185140 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.187076 (+ 1936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.188863 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.189809 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.189991 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:13.194823 (+ 4832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:13.194832 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:13.194844 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:13.194876 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:13.198137 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:13.198919 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:13.198926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:13.198932 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:13.199081 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:13.199464 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:13.199469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:13.199472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:13.199982 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:13.200290 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:13.200815 (+ 525us) server_negotiation.cc:299] Negotiation successful
0928 08:36:13.201266 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":392,"threads_started":1}
I20260928 08:36:13.203302 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:13.186166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:45883 (local address 127.0.0.1:49918)
0928 08:36:13.187081 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:13.187119 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:36:13.187308 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:36:13.187726 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:36:13.187733 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:36:13.188597 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:36:13.188991 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:13.189002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.190398 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.190405 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:13.192151 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:13.192172 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.193073 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.193084 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:13.193274 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:13.194834 (+ 1560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:36:13.194857 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:36:13.197426 (+ 2569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:36:13.200536 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:13.200548 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:13.200557 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:13.201085 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:13.201538 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:13.201542 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:13.201546 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:13.201675 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:13.202285 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:36:13.202291 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:36:13.202449 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 08:36:13.202616 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 08:36:13.202956 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"spinlock_wait_cycles":15616,"thread_start_us":566,"threads_started":1}
I20260928 08:36:13.203439 6061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:13.186136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49918 (local address 127.15.134.254:45883)
0928 08:36:13.186755 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:13.186763 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:13.187406 (+ 643us) server_negotiation.cc:407] Connection header received
0928 08:36:13.187466 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:13.187471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:13.187539 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:13.187629 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:13.189135 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.190227 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.192330 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.192833 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.192928 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:13.197636 (+ 4708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:13.197642 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:13.197645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:13.197682 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:13.200393 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:13.201229 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:13.201234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:13.201237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:13.201404 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:13.201805 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:13.201810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:13.201814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:13.202137 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:13.202309 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:13.202778 (+ 469us) server_negotiation.cc:299] Negotiation successful
0928 08:36:13.203020 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":285,"threads_started":1}
I20260928 08:36:13.204988 6057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:36:13 dist-test-slave-qp2n krb5kdc[5824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584573, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.252@KRBTEST.COM for kudu/127.15.134.253@KRBTEST.COM
Sep 28 08:36:13 dist-test-slave-qp2n krb5kdc[5824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:36:13.219271 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:13.205831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:45467 (local address 127.0.0.1:54620)
0928 08:36:13.206003 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:13.206021 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:36:13.206122 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:36:13.206528 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:36:13.206533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:36:13.206946 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:36:13.207140 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:13.207149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.208440 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.208448 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:13.209604 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:13.209617 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.210609 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.210615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:13.210807 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:13.211430 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:36:13.211452 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:36:13.213486 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:36:13.216895 (+ 3409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:13.216903 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:13.216908 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:13.217420 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:13.217772 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:13.217776 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:13.217781 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:13.217950 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:13.218454 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:36:13.218459 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:36:13.218614 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 08:36:13.218737 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 08:36:13.218925 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 08:36:13.219642 5986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:13.205840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54620 (local address 127.15.134.253:45467)
0928 08:36:13.206201 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:13.206208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:13.206228 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:36:13.206289 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:13.206298 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:13.206391 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:13.206463 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:13.207292 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.208309 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.209764 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.210441 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.210578 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:13.213674 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:13.213680 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:13.213683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:13.213716 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:13.216754 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:13.217530 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:13.217534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:13.217537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:13.217658 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:13.218048 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:13.218051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:13.218062 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:13.218338 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:13.218480 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:13.218876 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 08:36:13.219100 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218}
I20260928 08:36:13.220599 6057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:36:13.222028 6061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:13.206032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49932 (local address 127.15.134.254:45883)
0928 08:36:13.206198 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:13.206203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:13.206222 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:36:13.206347 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:13.206351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:13.206414 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:13.206523 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:13.208004 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.209023 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.210584 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.211184 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.211335 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:13.215331 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:13.215336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:13.215339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:13.215370 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:13.218351 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:13.219077 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:13.219084 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:13.219088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:13.219275 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:13.219962 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:13.219968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:13.219972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:13.220241 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:13.220510 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:13.221439 (+ 929us) server_negotiation.cc:299] Negotiation successful
0928 08:36:13.221668 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 08:36:13 dist-test-slave-qp2n krb5kdc[5824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584573, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.252@KRBTEST.COM for kudu/127.15.134.252@KRBTEST.COM
Sep 28 08:36:13 dist-test-slave-qp2n krb5kdc[5824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.253@KRBTEST.COM
I20260928 08:36:13.235276 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:13.221438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:40233 (local address 127.0.0.1:48432)
0928 08:36:13.221711 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:13.221729 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:36:13.221839 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:36:13.222469 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:36:13.222477 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:36:13.222942 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:36:13.223104 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:13.223114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.224331 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.224336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:13.225806 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:13.225820 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.226739 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.226742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:13.226898 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:13.227461 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:36:13.227484 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:36:13.229878 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:36:13.233033 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:13.233037 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:13.233039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:13.233370 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:13.233763 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:13.233769 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:13.233773 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:13.233928 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:13.234465 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:36:13.234470 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:36:13.234599 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 08:36:13.234779 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 08:36:13.235007 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260928 08:36:13.235616 6060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:13.221407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48432 (local address 127.15.134.252:40233)
0928 08:36:13.221582 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:13.221587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:13.221844 (+ 257us) server_negotiation.cc:407] Connection header received
0928 08:36:13.221948 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:13.221957 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:13.222012 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:13.222081 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:13.223242 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.224212 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.225941 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.226582 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.226715 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:13.230047 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:13.230053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:13.230058 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:13.230094 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:13.232873 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:13.233478 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:13.233486 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:13.233490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:13.233643 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:13.234073 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:13.234078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:13.234081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:13.234354 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:13.234501 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:13.234877 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 08:36:13.235178 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":60}
I20260928 08:36:13.239369 5986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:13.223835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54624 (local address 127.15.134.253:45467)
0928 08:36:13.224013 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:13.224019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:13.224035 (+ 16us) server_negotiation.cc:407] Connection header received
0928 08:36:13.224077 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:13.224082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:13.224143 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:13.224254 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:13.225797 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.226859 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.228574 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.229019 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.229160 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:13.233561 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:13.233570 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:13.233576 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:13.233616 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:13.236829 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:13.237434 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:13.237442 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:13.237448 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:13.237622 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:13.237982 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:13.237993 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:13.237997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:13.238330 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:13.238600 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:13.238785 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 08:36:13.239010 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 08:36:13.241106 6057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39df30f6d0db40b48db404b394a9cdd9: Bootstrap starting.
I20260928 08:36:13.244041 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:36:13.244588 6057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39df30f6d0db40b48db404b394a9cdd9: Neither blocks nor log segments found. Creating new log.
I20260928 08:36:13.245949 6057 log.cc:824] T 00000000000000000000000000000000 P 39df30f6d0db40b48db404b394a9cdd9: Log is configured to *not* fsync() on all Append() calls
I20260928 08:36:13.248886 6057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39df30f6d0db40b48db404b394a9cdd9: No bootstrap required, opened a new log
I20260928 08:36:13.246778 5881 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "97182f293fae4db8bbe96f60774a8df0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49932
I20260928 08:36:13.255615 5906 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } attempt: 4
I20260928 08:36:13.256026 6057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39df30f6d0db40b48db404b394a9cdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97182f293fae4db8bbe96f60774a8df0" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45883 } } peers { permanent_uuid: "fcf02e5e6981450a912dcf5a764410b6" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 } } peers { permanent_uuid: "39df30f6d0db40b48db404b394a9cdd9" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } }
I20260928 08:36:13.256455 6057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39df30f6d0db40b48db404b394a9cdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39df30f6d0db40b48db404b394a9cdd9, State: Initialized, Role: FOLLOWER
I20260928 08:36:13.257086 6057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39df30f6d0db40b48db404b394a9cdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97182f293fae4db8bbe96f60774a8df0" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45883 } } peers { permanent_uuid: "fcf02e5e6981450a912dcf5a764410b6" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 } } peers { permanent_uuid: "39df30f6d0db40b48db404b394a9cdd9" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } }
I20260928 08:36:13.258131 6064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39df30f6d0db40b48db404b394a9cdd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97182f293fae4db8bbe96f60774a8df0" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45883 } } peers { permanent_uuid: "fcf02e5e6981450a912dcf5a764410b6" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 } } peers { permanent_uuid: "39df30f6d0db40b48db404b394a9cdd9" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } } }
I20260928 08:36:13.258574 6064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39df30f6d0db40b48db404b394a9cdd9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:36:13.259351 6057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39df30f6d0db40b48db404b394a9cdd9 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 08:36:13 dist-test-slave-qp2n krb5kdc[5824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584572, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for kudu/127.15.134.252@KRBTEST.COM
W20260928 08:36:13.269940 6075 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 39df30f6d0db40b48db404b394a9cdd9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:36:13.270133 6075 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:36:13.273626 5908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:13.256345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:40233 (local address 127.0.0.1:48444)
0928 08:36:13.256758 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:13.256777 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:36:13.256891 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:36:13.257295 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:36:13.257301 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:36:13.257895 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:36:13.258107 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:13.258122 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.259293 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.259300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:13.260831 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:13.260846 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.261742 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.261748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:13.261953 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:13.262649 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:36:13.262676 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:36:13.265549 (+ 2873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:36:13.270306 (+ 4757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:13.270314 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:13.270319 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:13.270861 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:13.271393 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:13.271398 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:13.271401 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:13.271611 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:13.272472 (+ 861us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:36:13.272479 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:36:13.272696 (+ 217us) client_negotiation.cc:769] Sending connection context
0928 08:36:13.272849 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 08:36:13.273163 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260928 08:36:13.274297 6060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:13.256456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48444 (local address 127.15.134.252:40233)
0928 08:36:13.256727 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:13.256734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:13.256852 (+ 118us) server_negotiation.cc:407] Connection header received
0928 08:36:13.257007 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:13.257012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:13.257119 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:13.257236 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:13.258255 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.259150 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.260967 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.261586 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.261737 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:13.265725 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:13.265732 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:13.265737 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:13.265777 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:13.269968 (+ 4191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:13.271014 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:13.271020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:13.271024 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:13.271253 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:13.271778 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:13.271784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:13.271788 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:13.272225 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:13.272401 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:13.273126 (+ 725us) server_negotiation.cc:299] Negotiation successful
0928 08:36:13.273541 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 08:36:13.267392 5989 master_runner.cc:186] Error getting master registration for 127.15.134.254:45883: OK, instance_id { permanent_uuid: "97182f293fae4db8bbe96f60774a8df0" instance_seqno: 1790584572116220 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 08:36:13.279624 5989 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 08:36:13.280356 5906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0: Bootstrap starting.
I20260928 08:36:13.283552 5906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0: Neither blocks nor log segments found. Creating new log.
I20260928 08:36:13.284657 5906 log.cc:824] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0: Log is configured to *not* fsync() on all Append() calls
I20260928 08:36:13.287516 5906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0: No bootstrap required, opened a new log
I20260928 08:36:13.292661 5906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97182f293fae4db8bbe96f60774a8df0" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45883 } } peers { permanent_uuid: "fcf02e5e6981450a912dcf5a764410b6" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 } } peers { permanent_uuid: "39df30f6d0db40b48db404b394a9cdd9" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } }
I20260928 08:36:13.293011 5906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97182f293fae4db8bbe96f60774a8df0, State: Initialized, Role: FOLLOWER
I20260928 08:36:13.293452 5906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97182f293fae4db8bbe96f60774a8df0" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45883 } } peers { permanent_uuid: "fcf02e5e6981450a912dcf5a764410b6" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 } } peers { permanent_uuid: "39df30f6d0db40b48db404b394a9cdd9" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } }
I20260928 08:36:13.294417 6078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97182f293fae4db8bbe96f60774a8df0" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45883 } } peers { permanent_uuid: "fcf02e5e6981450a912dcf5a764410b6" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 } } peers { permanent_uuid: "39df30f6d0db40b48db404b394a9cdd9" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } } }
I20260928 08:36:13.294842 6078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:36:13.295610 5906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:36:13.295750 5881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 08:36:13.295979 5881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:36:13.299949 5881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97182f293fae4db8bbe96f60774a8df0" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45883 } } peers { permanent_uuid: "fcf02e5e6981450a912dcf5a764410b6" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 } } peers { permanent_uuid: "39df30f6d0db40b48db404b394a9cdd9" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } }
I20260928 08:36:13.303536 6032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "97182f293fae4db8bbe96f60774a8df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "39df30f6d0db40b48db404b394a9cdd9"
I20260928 08:36:13.304171 6032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39df30f6d0db40b48db404b394a9cdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:36:13.309005 6032 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 39df30f6d0db40b48db404b394a9cdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97182f293fae4db8bbe96f60774a8df0 in term 1.
I20260928 08:36:13.309842 5856 leader_election.cc:302] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39df30f6d0db40b48db404b394a9cdd9, 97182f293fae4db8bbe96f60774a8df0; no voters:
I20260928 08:36:13.310632 5881 leader_election.cc:288] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0 [CANDIDATE]: Term 1 election: Requested vote from peers fcf02e5e6981450a912dcf5a764410b6 (127.15.134.253:45467), 39df30f6d0db40b48db404b394a9cdd9 (127.15.134.252:40233)
I20260928 08:36:13.310451 6078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0 [term 1 FOLLOWER]: Leader election won for term 1
W20260928 08:36:13.330420 6089 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:36:13.330690 6089 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:36:13.333550 6078 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0 [term 1 LEADER]: Becoming Leader. State: Replica: 97182f293fae4db8bbe96f60774a8df0, State: Running, Role: LEADER
I20260928 08:36:13.310418 5837 master_runner.cc:186] Error getting master registration for 127.15.134.253:45467: OK, instance_id { permanent_uuid: "fcf02e5e6981450a912dcf5a764410b6" instance_seqno: 1790584572635753 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 08:36:13.334261 5837 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 08:36:13.334424 6078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97182f293fae4db8bbe96f60774a8df0" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45883 } } peers { permanent_uuid: "fcf02e5e6981450a912dcf5a764410b6" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 } } peers { permanent_uuid: "39df30f6d0db40b48db404b394a9cdd9" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } }
I20260928 08:36:13.337270 6092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97182f293fae4db8bbe96f60774a8df0. Latest consensus state: current_term: 1 leader_uuid: "97182f293fae4db8bbe96f60774a8df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97182f293fae4db8bbe96f60774a8df0" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45883 } } peers { permanent_uuid: "fcf02e5e6981450a912dcf5a764410b6" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 } } peers { permanent_uuid: "39df30f6d0db40b48db404b394a9cdd9" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } } }
I20260928 08:36:13.339808 6092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0 [sys.catalog]: This master's current role is: LEADER
I20260928 08:36:13.341642 6097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:36:13.343919 6097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260928 08:36:13.355530 5860 tablet.cc:2369] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:36:13.365679 5981 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } attempt: 5
I20260928 08:36:13.374478 6032 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 39df30f6d0db40b48db404b394a9cdd9 [term 1 FOLLOWER]: Refusing update from remote peer 97182f293fae4db8bbe96f60774a8df0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:36:13.376844 6092 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39df30f6d0db40b48db404b394a9cdd9" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
Sep 28 08:36:13 dist-test-slave-qp2n krb5kdc[5824](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584572, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.253@KRBTEST.COM for kudu/127.15.134.252@KRBTEST.COM
I20260928 08:36:13.394605 5984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:13.366384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:40233 (local address 127.0.0.1:48474)
0928 08:36:13.366677 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:13.366697 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:36:13.366806 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:36:13.372256 (+ 5450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:36:13.372263 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:36:13.372927 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:36:13.373177 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:13.373195 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.375357 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.375363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:13.376930 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:13.376947 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.378081 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.378088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:13.378387 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:13.379161 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:36:13.379182 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:36:13.382343 (+ 3161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:36:13.390704 (+ 8361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:13.390712 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:13.390716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:13.391380 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:13.392205 (+ 825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:13.392212 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:13.392215 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:13.392456 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:13.393426 (+ 970us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:36:13.393433 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:36:13.393646 (+ 213us) client_negotiation.cc:769] Sending connection context
0928 08:36:13.393796 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 08:36:13.394139 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260928 08:36:13.396270 6102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:13.366563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48474 (local address 127.15.134.252:40233)
0928 08:36:13.371813 (+ 5250us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:13.371821 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:13.371848 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:36:13.371920 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:13.371926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:13.372024 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:13.373362 (+ 1338us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:13.373744 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.375168 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.377151 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.377902 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.378793 (+ 891us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:13.382687 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:13.382699 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:13.382704 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:13.382752 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:13.387458 (+ 4706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:13.391815 (+ 4357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:13.391821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:13.391825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:13.392035 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:13.392663 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:13.392670 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:13.392674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:13.393072 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:13.395345 (+ 2273us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:13.395486 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 08:36:13.395765 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5081,"thread_start_us":6481,"threads_started":1}
I20260928 08:36:13.399078 6064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39df30f6d0db40b48db404b394a9cdd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97182f293fae4db8bbe96f60774a8df0. Latest consensus state: current_term: 1 leader_uuid: "97182f293fae4db8bbe96f60774a8df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97182f293fae4db8bbe96f60774a8df0" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45883 } } peers { permanent_uuid: "fcf02e5e6981450a912dcf5a764410b6" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 } } peers { permanent_uuid: "39df30f6d0db40b48db404b394a9cdd9" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } } }
I20260928 08:36:13.399425 6064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39df30f6d0db40b48db404b394a9cdd9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:36:13.401655 6099 mvcc.cc:205] Tried to move back new op lower bound from 7334234412522569728 to 7334234412382924800. Current Snapshot: MvccSnapshot[applied={T|T < 7334234412522569728}]
I20260928 08:36:13.404083 5981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcf02e5e6981450a912dcf5a764410b6: Bootstrap starting.
I20260928 08:36:13.414561 6061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:13.340866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49936 (local address 127.15.134.254:45883)
0928 08:36:13.341104 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:13.341110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:13.359680 (+ 18570us) server_negotiation.cc:407] Connection header received
0928 08:36:13.360523 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:13.360531 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:13.360648 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:13.360737 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:13.375345 (+ 14608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.376588 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.394318 (+ 17730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.395048 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.395165 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:13.406360 (+ 11195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:13.406368 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:13.406374 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:13.406419 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:13.410900 (+ 4481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:13.411961 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:13.411968 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:13.411972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:13.412135 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:13.412640 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:13.412646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:13.412650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:13.413020 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:13.413305 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:13.413923 (+ 618us) server_negotiation.cc:299] Negotiation successful
0928 08:36:13.414163 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 08:36:13.435344 5981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcf02e5e6981450a912dcf5a764410b6: Neither blocks nor log segments found. Creating new log.
I20260928 08:36:13.440725 5986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:13.340955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54638 (local address 127.15.134.253:45467)
0928 08:36:13.341121 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:13.341128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:13.361610 (+ 20482us) server_negotiation.cc:407] Connection header received
0928 08:36:13.361943 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:13.361950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:13.362066 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:13.362159 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:13.376728 (+ 14569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.378101 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.384603 (+ 6502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.385447 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.385582 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:13.426789 (+ 41207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:13.426799 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:13.426808 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:13.426871 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:13.431974 (+ 5103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:13.435695 (+ 3721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:13.435707 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:13.435714 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:13.435951 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:13.436711 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:13.436720 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:13.436726 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:13.437255 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:13.437603 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:13.439613 (+ 2010us) server_negotiation.cc:299] Negotiation successful
0928 08:36:13.439989 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:36:13.446475 5981 log.cc:824] T 00000000000000000000000000000000 P fcf02e5e6981450a912dcf5a764410b6: Log is configured to *not* fsync() on all Append() calls
I20260928 08:36:13.455406 6092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "97182f293fae4db8bbe96f60774a8df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97182f293fae4db8bbe96f60774a8df0" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45883 } } peers { permanent_uuid: "fcf02e5e6981450a912dcf5a764410b6" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 } } peers { permanent_uuid: "39df30f6d0db40b48db404b394a9cdd9" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } } }
I20260928 08:36:13.455729 6092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0 [sys.catalog]: This master's current role is: LEADER
I20260928 08:36:13.463086 6060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:13.342032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48460 (local address 127.15.134.252:40233)
0928 08:36:13.342209 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:13.342216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:13.373288 (+ 31072us) server_negotiation.cc:407] Connection header received
0928 08:36:13.376209 (+ 2921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:13.376215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:13.376296 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:13.376365 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:13.386061 (+ 9696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.387401 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.420347 (+ 32946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:13.421501 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:13.421655 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:13.446995 (+ 25340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:13.447009 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:13.447018 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:13.447153 (+ 135us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:13.453494 (+ 6341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:13.459300 (+ 5806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:13.459310 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:13.459316 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:13.459785 (+ 469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:13.460738 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:13.460744 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:13.460747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:13.461149 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:13.461583 (+ 434us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:13.462256 (+ 673us) server_negotiation.cc:299] Negotiation successful
0928 08:36:13.462612 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 08:36:13.465508 6097 catalog_manager.cc:1347] Generated new cluster ID: 3feecf57bc2e4a3ebcc98d322bfd173c
I20260928 08:36:13.465631 6097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:36:13.472901 5981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcf02e5e6981450a912dcf5a764410b6: No bootstrap required, opened a new log
I20260928 08:36:13.486478 6064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39df30f6d0db40b48db404b394a9cdd9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "97182f293fae4db8bbe96f60774a8df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97182f293fae4db8bbe96f60774a8df0" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45883 } } peers { permanent_uuid: "fcf02e5e6981450a912dcf5a764410b6" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 } } peers { permanent_uuid: "39df30f6d0db40b48db404b394a9cdd9" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } } }
I20260928 08:36:13.486969 6064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39df30f6d0db40b48db404b394a9cdd9 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:36:13.493149 5981 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcf02e5e6981450a912dcf5a764410b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97182f293fae4db8bbe96f60774a8df0" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45883 } } peers { permanent_uuid: "fcf02e5e6981450a912dcf5a764410b6" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 } } peers { permanent_uuid: "39df30f6d0db40b48db404b394a9cdd9" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } }
I20260928 08:36:13.493729 5981 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcf02e5e6981450a912dcf5a764410b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcf02e5e6981450a912dcf5a764410b6, State: Initialized, Role: FOLLOWER
I20260928 08:36:13.494357 5981 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcf02e5e6981450a912dcf5a764410b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97182f293fae4db8bbe96f60774a8df0" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45883 } } peers { permanent_uuid: "fcf02e5e6981450a912dcf5a764410b6" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 } } peers { permanent_uuid: "39df30f6d0db40b48db404b394a9cdd9" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } }
I20260928 08:36:13.501597 6108 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcf02e5e6981450a912dcf5a764410b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97182f293fae4db8bbe96f60774a8df0" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45883 } } peers { permanent_uuid: "fcf02e5e6981450a912dcf5a764410b6" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 } } peers { permanent_uuid: "39df30f6d0db40b48db404b394a9cdd9" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } } }
I20260928 08:36:13.504232 6108 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcf02e5e6981450a912dcf5a764410b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:36:13.504124 5956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "97182f293fae4db8bbe96f60774a8df0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fcf02e5e6981450a912dcf5a764410b6"
I20260928 08:36:13.504081 5981 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcf02e5e6981450a912dcf5a764410b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:36:13.504484 5955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcf02e5e6981450a912dcf5a764410b6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:36:13.514556 5955 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fcf02e5e6981450a912dcf5a764410b6 [term 1 FOLLOWER]: Refusing update from remote peer 97182f293fae4db8bbe96f60774a8df0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:36:13.515861 6092 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcf02e5e6981450a912dcf5a764410b6" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:36:13.518492 5956 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fcf02e5e6981450a912dcf5a764410b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97182f293fae4db8bbe96f60774a8df0 in term 1.
I20260928 08:36:13.528424 6097 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:36:13.530576 6097 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:36:13.549162 6097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0: Generated new TSK 0
I20260928 08:36:13.550395 6097 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:36:13.555496 6108 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcf02e5e6981450a912dcf5a764410b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97182f293fae4db8bbe96f60774a8df0. Latest consensus state: current_term: 1 leader_uuid: "97182f293fae4db8bbe96f60774a8df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97182f293fae4db8bbe96f60774a8df0" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45883 } } peers { permanent_uuid: "fcf02e5e6981450a912dcf5a764410b6" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 } } peers { permanent_uuid: "39df30f6d0db40b48db404b394a9cdd9" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } } }
I20260928 08:36:13.555846 6108 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcf02e5e6981450a912dcf5a764410b6 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:36:13.555913 6092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "97182f293fae4db8bbe96f60774a8df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97182f293fae4db8bbe96f60774a8df0" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45883 } } peers { permanent_uuid: "fcf02e5e6981450a912dcf5a764410b6" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 } } peers { permanent_uuid: "39df30f6d0db40b48db404b394a9cdd9" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } } }
I20260928 08:36:13.556191 6092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97182f293fae4db8bbe96f60774a8df0 [sys.catalog]: This master's current role is: LEADER
I20260928 08:36:13.557274 6121 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcf02e5e6981450a912dcf5a764410b6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "97182f293fae4db8bbe96f60774a8df0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97182f293fae4db8bbe96f60774a8df0" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 45883 } } peers { permanent_uuid: "fcf02e5e6981450a912dcf5a764410b6" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 45467 } } peers { permanent_uuid: "39df30f6d0db40b48db404b394a9cdd9" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 40233 } } }
I20260928 08:36:13.557788 6123 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 08:36:13.558248 6121 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcf02e5e6981450a912dcf5a764410b6 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 08:36:13.641814 6061 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:36:13.629120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49944 (local address 127.15.134.254:45883)
0928 08:36:13.629561 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:13.629569 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:13.639485 (+ 9916us) server_negotiation.cc:407] Connection header received
0928 08:36:13.639979 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:13.639986 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:13.640095 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:13.640181 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:13.641572 (+ 1391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49944: BlockingRecv error: recv got EOF from 127.0.0.1:49944 (error 108)
Metrics: {"server-negotiator.queue_time_us":269}
W20260928 08:36:13.644258 6060 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:36:13.629312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48476 (local address 127.15.134.252:40233)
0928 08:36:13.629733 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:13.629739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:13.642594 (+ 12855us) server_negotiation.cc:407] Connection header received
0928 08:36:13.642831 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:13.642839 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:13.642925 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:13.643001 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:13.643973 (+ 972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48476: BlockingRecv error: recv got EOF from 127.0.0.1:48476 (error 108)
Metrics: {"server-negotiator.queue_time_us":306}
W20260928 08:36:13.647178 5986 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:36:13.634975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54646 (local address 127.15.134.253:45467)
0928 08:36:13.635167 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:13.635173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:13.644737 (+ 9564us) server_negotiation.cc:407] Connection header received
0928 08:36:13.645687 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:13.645703 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:13.645793 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:13.645859 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:13.646952 (+ 1093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54646: BlockingRecv error: recv got EOF from 127.0.0.1:54646 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 08:36:13.687601 5986 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:36:13.685220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54660 (local address 127.15.134.253:45467)
0928 08:36:13.685462 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:13.685472 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:13.686000 (+ 528us) server_negotiation.cc:407] Connection header received
0928 08:36:13.686277 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:13.686282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:13.686386 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:13.686523 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:13.687310 (+ 787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54660: BlockingRecv error: recv got EOF from 127.0.0.1:54660 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 08:36:13.698069 5986 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:36:13.691852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54676 (local address 127.15.134.253:45467)
0928 08:36:13.692016 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:13.692026 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:13.696288 (+ 4262us) server_negotiation.cc:407] Connection header received
0928 08:36:13.696532 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:13.696538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:13.696826 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:13.696950 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:13.697837 (+ 887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54676: BlockingRecv error: recv got EOF from 127.0.0.1:54676 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 08:36:13.705579 5986 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:36:13.703387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54690 (local address 127.15.134.253:45467)
0928 08:36:13.703544 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:13.703549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:13.704183 (+ 634us) server_negotiation.cc:407] Connection header received
0928 08:36:13.704408 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:13.704413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:13.704492 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:13.704592 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:13.705341 (+ 749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54690: BlockingRecv error: recv got EOF from 127.0.0.1:54690 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 08:36:13.715801 5986 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:36:13.713414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54698 (local address 127.15.134.253:45467)
0928 08:36:13.713690 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:13.713701 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:13.714294 (+ 593us) server_negotiation.cc:407] Connection header received
0928 08:36:13.714533 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:13.714538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:13.714619 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:13.714743 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:13.715585 (+ 842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54698: BlockingRecv error: recv got EOF from 127.0.0.1:54698 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
W20260928 08:36:13.730095 5986 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:36:13.727757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54710 (local address 127.15.134.253:45467)
0928 08:36:13.727995 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:13.728005 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:13.728572 (+ 567us) server_negotiation.cc:407] Connection header received
0928 08:36:13.728848 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:13.728860 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:13.728934 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:13.729042 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:13.729852 (+ 810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54710: BlockingRecv error: recv got EOF from 127.0.0.1:54710 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260928 08:36:13.751849 5986 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:36:13.749598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54716 (local address 127.15.134.253:45467)
0928 08:36:13.749840 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:13.749850 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:13.750401 (+ 551us) server_negotiation.cc:407] Connection header received
0928 08:36:13.750630 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:13.750635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:13.750723 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:13.750833 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:13.751599 (+ 766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54716: BlockingRecv error: recv got EOF from 127.0.0.1:54716 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260928 08:36:13.789602 5986 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:36:13.787282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54720 (local address 127.15.134.253:45467)
0928 08:36:13.787520 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:13.787530 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:13.788087 (+ 557us) server_negotiation.cc:407] Connection header received
0928 08:36:13.788359 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:13.788365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:13.788440 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:13.788550 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:13.789333 (+ 783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54720: BlockingRecv error: recv got EOF from 127.0.0.1:54720 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 08:36:13.859128 5986 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:36:13.856872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54728 (local address 127.15.134.253:45467)
0928 08:36:13.857130 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:13.857136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:13.857724 (+ 588us) server_negotiation.cc:407] Connection header received
0928 08:36:13.857959 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:13.857964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:13.858040 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:13.858153 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:13.858909 (+ 756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54728: BlockingRecv error: recv got EOF from 127.0.0.1:54728 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260928 08:36:13.996459 5986 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:36:13.994456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54734 (local address 127.15.134.253:45467)
0928 08:36:13.994748 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:13.994754 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:13.995009 (+ 255us) server_negotiation.cc:407] Connection header received
0928 08:36:13.995168 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:13.995172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:13.995262 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:13.995349 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:13.996222 (+ 873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54734: BlockingRecv error: recv got EOF from 127.0.0.1:54734 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260928 08:36:14.259392 5986 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:36:14.256915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54742 (local address 127.15.134.253:45467)
0928 08:36:14.257202 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:14.257211 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:14.257487 (+ 276us) server_negotiation.cc:407] Connection header received
0928 08:36:14.257787 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:14.257791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:14.257871 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:14.257974 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:14.259153 (+ 1179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54742: BlockingRecv error: recv got EOF from 127.0.0.1:54742 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
I20260928 08:36:14.272706 6075 catalog_manager.cc:1259] Loaded cluster ID: 3feecf57bc2e4a3ebcc98d322bfd173c
I20260928 08:36:14.272818 6075 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 39df30f6d0db40b48db404b394a9cdd9: loading cluster ID for follower catalog manager: success
I20260928 08:36:14.275676 6075 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 39df30f6d0db40b48db404b394a9cdd9: acquiring CA information for follower catalog manager: success
I20260928 08:36:14.277700 6075 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 39df30f6d0db40b48db404b394a9cdd9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:36:14.577165 6123 catalog_manager.cc:1259] Loaded cluster ID: 3feecf57bc2e4a3ebcc98d322bfd173c
I20260928 08:36:14.577340 6123 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fcf02e5e6981450a912dcf5a764410b6: loading cluster ID for follower catalog manager: success
I20260928 08:36:14.580278 6123 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fcf02e5e6981450a912dcf5a764410b6: acquiring CA information for follower catalog manager: success
I20260928 08:36:14.582432 6123 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fcf02e5e6981450a912dcf5a764410b6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:36:14.784381 6191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:14.775827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54746 (local address 127.15.134.253:45467)
0928 08:36:14.777691 (+ 1864us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:14.777696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:14.777715 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:36:14.777786 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:14.777790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:14.777928 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:14.778039 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:36:14.778626 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:14.779701 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:14.781841 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:14.782321 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:14.782424 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:14.783076 (+ 652us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:36:14.783643 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:36:14.783766 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:14.783930 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 08:36:14.784005 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1669,"thread_start_us":1568,"threads_started":1}
I20260928 08:36:14.798877 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:14.791845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48490 (local address 127.15.134.252:40233)
0928 08:36:14.792562 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:14.792566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:14.792584 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:36:14.792729 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:14.792733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:14.792894 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:14.793076 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:36:14.793535 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:14.794320 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:14.796329 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:14.796899 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:14.797069 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:14.797643 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:36:14.798219 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:36:14.798340 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:14.798481 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 08:36:14.798558 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":512,"threads_started":1}
I20260928 08:36:14.822829 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:14.808001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48492 (local address 127.15.134.252:40233)
0928 08:36:14.815645 (+ 7644us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:14.815654 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:14.815679 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:36:14.815824 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:14.815832 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:14.815965 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:14.816071 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:36:14.816878 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:14.818052 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:14.820390 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:14.821141 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:14.821253 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:14.821949 (+ 696us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:36:14.822098 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:36:14.822168 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:14.822399 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 08:36:14.822488 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5728}
I20260928 08:36:14.825214 6191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:14.807535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54754 (local address 127.15.134.253:45467)
0928 08:36:14.807779 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:14.807786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:14.818391 (+ 10605us) server_negotiation.cc:407] Connection header received
0928 08:36:14.818697 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:14.818704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:14.818799 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:14.818868 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:36:14.819687 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:14.820884 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:14.823148 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:14.823812 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:14.823915 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:14.824398 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:36:14.824547 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:36:14.824654 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:14.824856 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 08:36:14.824945 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 08:36:14.826392 6202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:14.807456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49952 (local address 127.15.134.254:45883)
0928 08:36:14.811024 (+ 3568us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:14.811031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:14.817621 (+ 6590us) server_negotiation.cc:407] Connection header received
0928 08:36:14.817838 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:14.817844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:14.818064 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:14.818270 (+ 206us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:36:14.819152 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:14.820218 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:14.822525 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:14.823464 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:14.823598 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:14.825098 (+ 1500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:36:14.825662 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:36:14.825767 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:14.825998 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 08:36:14.826106 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3411,"thread_start_us":3302,"threads_started":1}
I20260928 08:36:14.833151 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 5837
I20260928 08:36:14.834568 5901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:36:14.951653 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 5837
I20260928 08:36:14.973726 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 5913
I20260928 08:36:14.974889 5976 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:36:15.083350 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 5913
I20260928 08:36:15.106024 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 5989
I20260928 08:36:15.107151 6052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:36:15.214836 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 5989
2026-09-28T08:36:15Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4941 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:36:15 dist-test-slave-qp2n krb5kdc[6211](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:36:15 dist-test-slave-qp2n krb5kdc[6211](info): set up 2 sockets
Sep 28 08:36:15 dist-test-slave-qp2n krb5kdc[6211](info): commencing operation
krb5kdc: starting...
W20260928 08:36:16.382879 15899 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/krb5kdc/test-user.keytab.
Sep 28 08:36:16 dist-test-slave-qp2n krb5kdc[6211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:36:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:36:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.15.134.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
Entry for principal kudu/127.15.134.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab.
I20260928 08:36:16.671767 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:38047
--webserver_interface=127.15.134.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:35953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:38047,127.15.134.253:36201,127.15.134.252:44707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:36:17.017477 6224 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 08:36:17.017787 6224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:36:17.017859 6224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:36:17.017943 6224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:36:17.025091 6224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:36:17.025192 6224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:36:17.025249 6224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:36:17.025291 6224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:36:17.025352 6224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:36:17.035271 6224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:35953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:38047,127.15.134.253:36201,127.15.134.252:44707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.254.keytab
--principal=kudu/127.15.134.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.254:38047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.15.134.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:36:17.036540 6224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:36:17.037997 6224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:36:17.050276 6230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:36:17.050339 6229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:36:17.050364 6232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:36:17.051422 6224 server_base.cc:1034] running on GCE node
I20260928 08:36:17.052269 6224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:36:17.054009 6224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:36:17.055249 6224 hybrid_clock.cc:648] HybridClock initialized: now 1790584577055190 us; error 30 us; skew 500 ppm
I20260928 08:36:17.055626 6224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:36:17 dist-test-slave-qp2n krb5kdc[6211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584577, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:36:17.059064 6224 init.cc:377] Logged in from keytab as kudu/127.15.134.254@KRBTEST.COM (short username kudu)
I20260928 08:36:17.060756 6224 webserver.cc:460] Webserver started at http://127.15.134.254:36777/ using document root <none> and password file <none>
I20260928 08:36:17.061564 6224 fs_manager.cc:359] Metadata directory not provided
I20260928 08:36:17.061708 6224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:36:17.062043 6224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:36:17.065205 6224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-0/data/instance:
uuid: "d9a2100921884f22b34d30cf770baf46"
format_stamp: "Formatted at 2026-09-28 08:36:17 on dist-test-slave-qp2n"
server_key: "f26fe30efffc2c7acd1d88a1d59c09fc"
server_key_iv: "6ed67c0f40eb9e659adf8f03faaa6974"
server_key_version: "encryptionkey@0"
I20260928 08:36:17.066037 6224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-0/wal/instance:
uuid: "d9a2100921884f22b34d30cf770baf46"
format_stamp: "Formatted at 2026-09-28 08:36:17 on dist-test-slave-qp2n"
server_key: "f26fe30efffc2c7acd1d88a1d59c09fc"
server_key_iv: "6ed67c0f40eb9e659adf8f03faaa6974"
server_key_version: "encryptionkey@0"
I20260928 08:36:17.071436 6224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 08:36:17.075120 6239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:36:17.076539 6224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 08:36:17.076781 6224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-0/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-0/wal
uuid: "d9a2100921884f22b34d30cf770baf46"
format_stamp: "Formatted at 2026-09-28 08:36:17 on dist-test-slave-qp2n"
server_key: "f26fe30efffc2c7acd1d88a1d59c09fc"
server_key_iv: "6ed67c0f40eb9e659adf8f03faaa6974"
server_key_version: "encryptionkey@0"
I20260928 08:36:17.077024 6224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:36:17.105268 6224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:36:17.109869 6224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:36:17.110149 6224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:36:17.129112 6224 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.254:38047
I20260928 08:36:17.129132 6290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.254:38047 every 8 connection(s)
I20260928 08:36:17.130496 6224 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-0/data/info.pb
I20260928 08:36:17.134302 6291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:36:17.138830 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 6224
I20260928 08:36:17.139021 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-0/wal/instance
I20260928 08:36:17.139479 15899 external_mini_cluster.cc:1428] Setting key d845c924d5d60650e737a28bffb623d6
I20260928 08:36:17.141177 6291 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38047 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:36:17 dist-test-slave-qp2n krb5kdc[6211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584577, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:36:17.166853 6294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.144593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36880 (local address 127.15.134.254:38047)
0928 08:36:17.147119 (+ 2526us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:17.147130 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:17.147185 (+ 55us) server_negotiation.cc:407] Connection header received
0928 08:36:17.148556 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:17.148568 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:17.148902 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:17.149074 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:17.150603 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:17.152672 (+ 2069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:17.154146 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:17.155156 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:17.155360 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:17.159317 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:17.159325 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:17.159339 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:17.159373 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:17.162260 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:17.163236 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:17.163243 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:17.163245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:17.163390 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:17.163901 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:17.163904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:17.163906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:17.164421 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:17.164572 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:17.165339 (+ 767us) server_negotiation.cc:299] Negotiation successful
0928 08:36:17.165703 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":2239,"spinlock_wait_cycles":21760,"thread_start_us":1988,"threads_started":1}
I20260928 08:36:17.166888 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.144593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:38047 (local address 127.0.0.1:36880)
0928 08:36:17.145661 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.145735 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0928 08:36:17.148128 (+ 2393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:36:17.149170 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:36:17.149180 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:36:17.150033 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:36:17.150376 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:17.150394 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:17.152834 (+ 2440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:17.152838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:17.153984 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:17.154001 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:17.155385 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:17.155390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:17.155545 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:17.156274 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:36:17.156303 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:36:17.159068 (+ 2765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:36:17.162441 (+ 3373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:17.162458 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:17.162475 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:17.163093 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:17.163523 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:17.163530 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:17.163537 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:17.163781 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:17.164592 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:36:17.164603 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:36:17.164923 (+ 320us) client_negotiation.cc:769] Sending connection context
0928 08:36:17.165242 (+ 319us) client_negotiation.cc:240] Negotiation successful
0928 08:36:17.165879 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":112,"thread_start_us":374,"threads_started":1}
WARNING: no policy specified for kudu/127.15.134.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.253@KRBTEST.COM" created.
I20260928 08:36:17.170284 6291 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:36:17.171536 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.170986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:36201 (local address 127.0.0.1:40030)
0928 08:36:17.171204 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.171377 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:36201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 08:36:17.171835 6243 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.134.253:36201: connect: Connection refused (error 111)
I20260928 08:36:17.173571 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.173108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:36201 (local address 127.0.0.1:40036)
0928 08:36:17.173302 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.173393 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:36201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 08:36:17.173805 6291 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.253:36201: Network error: Client connection negotiation failed: client connection to 127.15.134.253:36201: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.253.keytab.
Entry for principal kudu/127.15.134.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.253.keytab.
I20260928 08:36:17.200794 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.253:36201
--webserver_interface=127.15.134.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.253.keytab
--principal=kudu/127.15.134.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:35953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:38047,127.15.134.253:36201,127.15.134.252:44707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:36:17.223170 6291 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 } attempt: 1
I20260928 08:36:17.224360 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.223881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:36201 (local address 127.0.0.1:40048)
0928 08:36:17.224088 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.224228 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:36201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 08:36:17.225371 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.225020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:36201 (local address 127.0.0.1:40062)
0928 08:36:17.225182 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.225247 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:36201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 08:36:17.225556 6291 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.253:36201: Network error: Client connection negotiation failed: client connection to 127.15.134.253:36201: connect: Connection refused (error 111)
I20260928 08:36:17.293778 6291 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 } attempt: 2
I20260928 08:36:17.294921 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.294425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:36201 (local address 127.0.0.1:40076)
0928 08:36:17.294679 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.294765 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:36201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 08:36:17.295889 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.295538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:36201 (local address 127.0.0.1:40088)
0928 08:36:17.295705 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.295766 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:36201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 08:36:17.296059 6291 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.253:36201: Network error: Client connection negotiation failed: client connection to 127.15.134.253:36201: connect: Connection refused (error 111)
I20260928 08:36:17.387290 6291 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 } attempt: 3
I20260928 08:36:17.388536 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.388029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:36201 (local address 127.0.0.1:53962)
0928 08:36:17.388289 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.388386 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:36201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260928 08:36:17.389673 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.389302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:36201 (local address 127.0.0.1:53972)
0928 08:36:17.389451 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.389524 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:36201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 08:36:17.389869 6291 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.253:36201: Network error: Client connection negotiation failed: client connection to 127.15.134.253:36201: connect: Connection refused (error 111)
I20260928 08:36:17.533141 6291 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 } attempt: 4
I20260928 08:36:17.534402 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.533873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:36201 (local address 127.0.0.1:53984)
0928 08:36:17.534191 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.534272 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:36201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260928 08:36:17.535348 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.535018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:36201 (local address 127.0.0.1:53990)
0928 08:36:17.535160 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.535240 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.253:36201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 08:36:17.535557 6291 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.253:36201: Network error: Client connection negotiation failed: client connection to 127.15.134.253:36201: connect: Connection refused (error 111)
W20260928 08:36:17.541230 6298 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 08:36:17.541605 6298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:36:17.541731 6298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:36:17.541873 6298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:36:17.549299 6298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:36:17.549423 6298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:36:17.549546 6298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:36:17.549636 6298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:36:17.549733 6298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:36:17.559880 6298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:35953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:38047,127.15.134.253:36201,127.15.134.252:44707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.253.keytab
--principal=kudu/127.15.134.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.253:36201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.15.134.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:36:17.561360 6298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:36:17.562808 6298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:36:17.574976 6304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:36:17.574949 6306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:36:17.575435 6303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:36:17.576297 6298 server_base.cc:1034] running on GCE node
I20260928 08:36:17.577229 6298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:36:17.578644 6298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:36:17.579897 6298 hybrid_clock.cc:648] HybridClock initialized: now 1790584577579810 us; error 57 us; skew 500 ppm
I20260928 08:36:17.580303 6298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:36:17 dist-test-slave-qp2n krb5kdc[6211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584577, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:36:17.583791 6298 init.cc:377] Logged in from keytab as kudu/127.15.134.253@KRBTEST.COM (short username kudu)
I20260928 08:36:17.585440 6298 webserver.cc:460] Webserver started at http://127.15.134.253:37603/ using document root <none> and password file <none>
I20260928 08:36:17.586215 6298 fs_manager.cc:359] Metadata directory not provided
I20260928 08:36:17.586342 6298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:36:17.586694 6298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:36:17.589701 6298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-1/data/instance:
uuid: "36ce736df30d45989958f89c7229f69e"
format_stamp: "Formatted at 2026-09-28 08:36:17 on dist-test-slave-qp2n"
server_key: "77b1348f3e4a5ba0039cacfb0dbbb917"
server_key_iv: "5c0b7cd9329dacf48467e7307e5e368b"
server_key_version: "encryptionkey@0"
I20260928 08:36:17.590464 6298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-1/wal/instance:
uuid: "36ce736df30d45989958f89c7229f69e"
format_stamp: "Formatted at 2026-09-28 08:36:17 on dist-test-slave-qp2n"
server_key: "77b1348f3e4a5ba0039cacfb0dbbb917"
server_key_iv: "5c0b7cd9329dacf48467e7307e5e368b"
server_key_version: "encryptionkey@0"
I20260928 08:36:17.596184 6298 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:36:17.600000 6314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:36:17.601694 6298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:36:17.601854 6298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-1/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-1/wal
uuid: "36ce736df30d45989958f89c7229f69e"
format_stamp: "Formatted at 2026-09-28 08:36:17 on dist-test-slave-qp2n"
server_key: "77b1348f3e4a5ba0039cacfb0dbbb917"
server_key_iv: "5c0b7cd9329dacf48467e7307e5e368b"
server_key_version: "encryptionkey@0"
I20260928 08:36:17.602051 6298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:36:17.620115 6298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:36:17.624524 6298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:36:17.624782 6298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:36:17.643779 6298 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.253:36201
I20260928 08:36:17.643831 6365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.253:36201 every 8 connection(s)
I20260928 08:36:17.645249 6298 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-1/data/info.pb
I20260928 08:36:17.646104 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 6298
I20260928 08:36:17.646335 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-1/wal/instance
I20260928 08:36:17.646759 15899 external_mini_cluster.cc:1428] Setting key 5d9b1ea51460718a29b686d12791933d
I20260928 08:36:17.649152 6366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:36:17.655435 6366 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38047 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:36:17 dist-test-slave-qp2n krb5kdc[6211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584577, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.253@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.15.134.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.15.134.252@KRBTEST.COM" created.
I20260928 08:36:17.675122 6294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.658037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50362 (local address 127.15.134.254:38047)
0928 08:36:17.658220 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:17.658225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:17.659484 (+ 1259us) server_negotiation.cc:407] Connection header received
0928 08:36:17.661441 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:17.661447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:17.661538 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:17.661646 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:17.663070 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:17.663753 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:17.665298 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:17.665713 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:17.665850 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:17.669313 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:17.669318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:17.669321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:17.669352 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:17.672162 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:17.672931 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:17.672941 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:17.672943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:17.673054 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:17.673479 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:17.673484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:17.673489 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:17.673769 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:17.673926 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:17.674581 (+ 655us) server_negotiation.cc:299] Negotiation successful
0928 08:36:17.674802 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 08:36:17.676090 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.658605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:38047 (local address 127.0.0.1:50362)
0928 08:36:17.659330 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.659372 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:36:17.661155 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:36:17.661870 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:36:17.661879 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:36:17.662381 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:36:17.662901 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:17.662917 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:17.663914 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:17.663919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:17.665160 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:17.665172 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:17.665937 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:17.665944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:17.666176 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:17.666890 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:36:17.666912 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:36:17.669096 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:36:17.672299 (+ 3203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:17.672307 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:17.672322 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:17.672804 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:17.673180 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:17.673185 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:17.673188 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:17.673361 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:17.673905 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:36:17.673913 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:36:17.674234 (+ 321us) client_negotiation.cc:769] Sending connection context
0928 08:36:17.674484 (+ 250us) client_negotiation.cc:240] Negotiation successful
0928 08:36:17.674961 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":304,"threads_started":1}
I20260928 08:36:17.677733 6366 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:36:17 dist-test-slave-qp2n krb5kdc[6211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584577, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.253@KRBTEST.COM for kudu/127.15.134.253@KRBTEST.COM
I20260928 08:36:17.694115 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.678570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:36201 (local address 127.0.0.1:53996)
0928 08:36:17.678780 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.678799 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:36:17.678914 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:36:17.679935 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:36:17.679943 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:36:17.680505 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:36:17.680830 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:17.680845 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:17.682419 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:17.682427 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:17.683977 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:17.683994 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:17.684916 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:17.684928 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:17.685157 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:17.685800 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:36:17.685827 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:36:17.688337 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:36:17.691561 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:17.691566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:17.691570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:17.691952 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:17.692409 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:17.692436 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:17.692439 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:17.692577 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:17.693425 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:36:17.693429 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:36:17.693555 (+ 126us) client_negotiation.cc:769] Sending connection context
0928 08:36:17.693664 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 08:36:17.693862 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 08:36:17.694621 6370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.678654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53996 (local address 127.15.134.253:36201)
0928 08:36:17.679391 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:17.679402 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:17.679447 (+ 45us) server_negotiation.cc:407] Connection header received
0928 08:36:17.679507 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:17.679518 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:17.679753 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:17.679871 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:17.681021 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:17.682273 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:17.684132 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:17.684772 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:17.684867 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:17.688544 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:17.688554 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:17.688567 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:17.688595 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:17.691406 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:17.692114 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:17.692124 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:17.692130 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:17.692280 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:17.692726 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:17.692734 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:17.692740 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:17.693284 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:17.693535 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:17.693882 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 08:36:17.694179 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":400,"threads_started":1}
I20260928 08:36:17.697052 6366 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:36:17.698051 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.697635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:44707 (local address 127.0.0.1:55426)
0928 08:36:17.697793 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.697892 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 08:36:17.698316 6318 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
I20260928 08:36:17.699839 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.699478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:44707 (local address 127.0.0.1:55432)
0928 08:36:17.699669 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.699745 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 08:36:17.700038 6366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:44707: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.15.134.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.252.keytab.
Entry for principal kudu/127.15.134.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.252.keytab.
I20260928 08:36:17.705987 15899 external_mini_cluster.cc:1326] Running /tmp/dist-test-task0uZKZe/build/asan/bin/kudu
/tmp/dist-test-task0uZKZe/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.252:44707
--webserver_interface=127.15.134.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.252.keytab
--principal=kudu/127.15.134.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.15.134.212:35953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.134.254:38047,127.15.134.253:36201,127.15.134.252:44707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:36:17.749359 6366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } attempt: 1
I20260928 08:36:17.750717 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.750226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:44707 (local address 127.0.0.1:55436)
0928 08:36:17.750428 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.750556 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 08:36:17.751855 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.751476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:44707 (local address 127.0.0.1:55452)
0928 08:36:17.751663 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.751741 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 08:36:17.752156 6366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:44707: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
I20260928 08:36:17.820405 6366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } attempt: 2
I20260928 08:36:17.821575 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.821145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:44707 (local address 127.0.0.1:55468)
0928 08:36:17.821334 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.821425 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 08:36:17.822574 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.822284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:44707 (local address 127.0.0.1:55484)
0928 08:36:17.822405 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.822469 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 08:36:17.822847 6366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:44707: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
I20260928 08:36:17.834782 6291 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 } attempt: 5
Sep 28 08:36:17 dist-test-slave-qp2n krb5kdc[6211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584577, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for kudu/127.15.134.253@KRBTEST.COM
I20260928 08:36:17.848585 6370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.835777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54008 (local address 127.15.134.253:36201)
0928 08:36:17.835991 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:17.835999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:17.836019 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:36:17.836083 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:17.836086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:17.836171 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:17.836278 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:17.837179 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:17.837873 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:17.839020 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:17.839477 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:17.839621 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:17.843078 (+ 3457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:17.843082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:17.843086 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:17.843119 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:17.845962 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:17.846650 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:17.846655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:17.846659 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:17.846793 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:17.847180 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:17.847185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:17.847188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:17.847474 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:17.847628 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:17.848018 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 08:36:17.848251 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 08:36:17.848585 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.835518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:36201 (local address 127.0.0.1:54008)
0928 08:36:17.835745 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.835763 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:36:17.835899 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:36:17.836321 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:36:17.836328 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:36:17.836828 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:36:17.837012 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:17.837023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:17.838011 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:17.838017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:17.838868 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:17.838880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:17.839639 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:17.839646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:17.839913 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:17.840584 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:36:17.840603 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:36:17.842869 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:36:17.846103 (+ 3234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:17.846109 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:17.846113 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:17.846511 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:17.846912 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:17.846918 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:17.846922 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:17.847061 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:17.847593 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:36:17.847598 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:36:17.847765 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 08:36:17.847921 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 08:36:17.848222 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 08:36:17.849920 6291 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:36:17.851096 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.850564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:44707 (local address 127.0.0.1:55498)
0928 08:36:17.850765 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.850877 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 08:36:17.852116 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.851734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:44707 (local address 127.0.0.1:55512)
0928 08:36:17.851898 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.851991 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 08:36:17.852342 6291 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:44707: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
I20260928 08:36:17.903589 6291 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } attempt: 1
I20260928 08:36:17.904914 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.904327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:44707 (local address 127.0.0.1:55518)
0928 08:36:17.904568 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.904709 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 08:36:17.905941 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.905538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:44707 (local address 127.0.0.1:55522)
0928 08:36:17.905700 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.905783 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 08:36:17.906204 6291 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:44707: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
I20260928 08:36:17.914098 6366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } attempt: 3
I20260928 08:36:17.915146 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.914726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:44707 (local address 127.0.0.1:55532)
0928 08:36:17.914902 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.915003 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 08:36:17.916379 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.915841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:44707 (local address 127.0.0.1:55534)
0928 08:36:17.916094 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.916196 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260928 08:36:17.916600 6366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:44707: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
I20260928 08:36:17.974462 6291 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } attempt: 2
I20260928 08:36:17.975844 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.975232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:44707 (local address 127.0.0.1:55542)
0928 08:36:17.975479 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.975615 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 08:36:17.977026 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:17.976549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:44707 (local address 127.0.0.1:55550)
0928 08:36:17.976764 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:17.976868 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260928 08:36:17.977325 6291 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:44707: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
W20260928 08:36:18.054126 6373 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 08:36:18.054442 6373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:36:18.054514 6373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:36:18.054610 6373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 08:36:18.059870 6366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } attempt: 4
I20260928 08:36:18.061234 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:18.060598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:44707 (local address 127.0.0.1:55566)
0928 08:36:18.060928 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:18.061043 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 08:36:18.062290 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:18.061917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:44707 (local address 127.0.0.1:55568)
0928 08:36:18.062076 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:18.062162 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 08:36:18.062500 6366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:44707: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
W20260928 08:36:18.062884 6373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:36:18.062956 6373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:36:18.063011 6373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:36:18.063066 6373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:36:18.063127 6373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:36:18.073274 6373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.134.212:35953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.15.134.254:38047,127.15.134.253:36201,127.15.134.252:44707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/krb5kdc/kudu_127.15.134.252.keytab
--principal=kudu/127.15.134.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.134.252:44707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.15.134.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:36:18.074623 6373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:36:18.075958 6373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:36:18.083549 6291 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } attempt: 3
I20260928 08:36:18.084815 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:18.084211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:44707 (local address 127.0.0.1:55574)
0928 08:36:18.084457 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:18.084545 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 08:36:18.085901 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:18.085554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:44707 (local address 127.0.0.1:55588)
0928 08:36:18.085715 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:18.085801 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 08:36:18.086160 6291 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.15.134.252:44707: Network error: Client connection negotiation failed: client connection to 127.15.134.252:44707: connect: Connection refused (error 111)
W20260928 08:36:18.089648 6378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:36:18.089730 6379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:36:18.089689 6381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:36:18.090834 6373 server_base.cc:1034] running on GCE node
I20260928 08:36:18.091815 6373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:36:18.093616 6373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:36:18.094935 6373 hybrid_clock.cc:648] HybridClock initialized: now 1790584578094850 us; error 58 us; skew 500 ppm
I20260928 08:36:18.095368 6373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:36:18 dist-test-slave-qp2n krb5kdc[6211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584578, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:36:18.099236 6373 init.cc:377] Logged in from keytab as kudu/127.15.134.252@KRBTEST.COM (short username kudu)
I20260928 08:36:18.101089 6373 webserver.cc:460] Webserver started at http://127.15.134.252:46117/ using document root <none> and password file <none>
I20260928 08:36:18.101974 6373 fs_manager.cc:359] Metadata directory not provided
I20260928 08:36:18.102133 6373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:36:18.102505 6373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:36:18.106146 6373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-2/data/instance:
uuid: "04ba0230e2a94fa4ac1565f5fd59538c"
format_stamp: "Formatted at 2026-09-28 08:36:18 on dist-test-slave-qp2n"
server_key: "c3a57a4f66056bc73cbd3ae0338c8cb8"
server_key_iv: "e312c4e61503d781b932d77b17f6f021"
server_key_version: "encryptionkey@0"
I20260928 08:36:18.107148 6373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-2/wal/instance:
uuid: "04ba0230e2a94fa4ac1565f5fd59538c"
format_stamp: "Formatted at 2026-09-28 08:36:18 on dist-test-slave-qp2n"
server_key: "c3a57a4f66056bc73cbd3ae0338c8cb8"
server_key_iv: "e312c4e61503d781b932d77b17f6f021"
server_key_version: "encryptionkey@0"
I20260928 08:36:18.113404 6373 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260928 08:36:18.117724 6388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:36:18.119433 6373 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 08:36:18.119675 6373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-2/data,/tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-2/wal
uuid: "04ba0230e2a94fa4ac1565f5fd59538c"
format_stamp: "Formatted at 2026-09-28 08:36:18 on dist-test-slave-qp2n"
server_key: "c3a57a4f66056bc73cbd3ae0338c8cb8"
server_key_iv: "e312c4e61503d781b932d77b17f6f021"
server_key_version: "encryptionkey@0"
I20260928 08:36:18.119930 6373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:36:18.133460 6373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:36:18.138093 6373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:36:18.138383 6373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:36:18.158643 6373 rpc_server.cc:307] RPC server started. Bound to: 127.15.134.252:44707
I20260928 08:36:18.158663 6439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.134.252:44707 every 8 connection(s)
I20260928 08:36:18.160244 6373 server_base.cc:1166] Dumped server information to /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-2/data/info.pb
I20260928 08:36:18.163192 15899 external_mini_cluster.cc:1388] Started /tmp/dist-test-task0uZKZe/build/asan/bin/kudu as pid 6373
I20260928 08:36:18.163385 15899 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task0uZKZe/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790584205252592-15899-0/minicluster-data/master-2/wal/instance
I20260928 08:36:18.163825 15899 external_mini_cluster.cc:1428] Setting key e98f50654c2f41ed169710ca19a6a692
I20260928 08:36:18.164558 6440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:36:18.174978 6440 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38047 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:36:18 dist-test-slave-qp2n krb5kdc[6211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.252@KRBTEST.COM
Sep 28 08:36:18 dist-test-slave-qp2n krb5kdc[6211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584578, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.252@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
I20260928 08:36:18.189296 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:18.165781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55594 (local address 127.15.134.252:44707)
0928 08:36:18.166568 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:18.166585 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:18.166640 (+ 55us) server_negotiation.cc:407] Connection header received
0928 08:36:18.168373 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:18.168404 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:18.169012 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:18.169481 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:18.171029 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.173224 (+ 2195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.174663 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.175852 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.176061 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:18.180753 (+ 4692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:18.180767 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:18.180787 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:18.180838 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:18.184570 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:18.185463 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:18.185478 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:18.185485 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:18.185670 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:18.186082 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:18.186087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:18.186092 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:18.186684 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:18.187008 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:18.187446 (+ 438us) server_negotiation.cc:299] Negotiation successful
0928 08:36:18.187973 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":315,"threads_started":1}
I20260928 08:36:18.195128 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:18.178082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.254:38047 (local address 127.0.0.1:50378)
0928 08:36:18.179065 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:18.179114 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 08:36:18.179298 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:36:18.179758 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:36:18.179764 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:36:18.180394 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:36:18.180826 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:18.180838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.182249 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.182256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:18.184063 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:18.184078 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.184919 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.184923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:18.185051 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:18.185689 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:36:18.185708 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:36:18.188287 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:36:18.192074 (+ 3787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:18.192083 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:18.192088 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:18.192764 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:18.193195 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:18.193202 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:18.193206 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:18.193350 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:18.193988 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:36:18.193996 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:36:18.194188 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 08:36:18.194388 (+ 200us) client_negotiation.cc:240] Negotiation successful
0928 08:36:18.194707 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":575,"threads_started":1}
I20260928 08:36:18.195169 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:18.178105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50378 (local address 127.15.134.254:38047)
0928 08:36:18.178691 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:18.178700 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:18.179238 (+ 538us) server_negotiation.cc:407] Connection header received
0928 08:36:18.179435 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:18.179442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:18.179533 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:18.179664 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:18.180990 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.182074 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.184234 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.184735 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.184884 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:18.188515 (+ 3631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:18.188521 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:18.188524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:18.188561 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:18.191925 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:18.192910 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:18.192914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:18.192916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:18.193064 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:18.193467 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:18.193472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:18.193477 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:18.193843 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:18.194011 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:18.194494 (+ 483us) server_negotiation.cc:299] Negotiation successful
0928 08:36:18.194717 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":429,"thread_start_us":275,"threads_started":1}
I20260928 08:36:18.196695 6440 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:36:18 dist-test-slave-qp2n krb5kdc[6211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.254@KRBTEST.COM
Sep 28 08:36:18 dist-test-slave-qp2n krb5kdc[6211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584578, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.252@KRBTEST.COM for kudu/127.15.134.253@KRBTEST.COM
I20260928 08:36:18.210321 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:18.192396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50390 (local address 127.15.134.254:38047)
0928 08:36:18.193265 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:18.193269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:18.193291 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:36:18.193334 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:18.193338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:18.193392 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:18.193529 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:18.194941 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.196154 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.198124 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.198639 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.198797 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:18.203540 (+ 4743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:18.203546 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:18.203549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:18.203581 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:18.207275 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:18.207988 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:18.207994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:18.207998 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:18.208144 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:18.208580 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:18.208586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:18.208589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:18.208934 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:18.209223 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:18.209587 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 08:36:18.209834 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":724,"thread_start_us":486,"threads_started":1}
I20260928 08:36:18.210387 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:18.197560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.253:36201 (local address 127.0.0.1:54014)
0928 08:36:18.197757 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:18.197779 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:36:18.197907 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:36:18.198218 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:36:18.198223 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:36:18.198602 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:36:18.198765 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:18.198773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.199717 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.199722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:18.200595 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:18.200606 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.201369 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.201374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:18.201566 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:18.202098 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:36:18.202126 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:36:18.204108 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:36:18.208011 (+ 3903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:18.208014 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:18.208017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:18.208360 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:18.208806 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:18.208812 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:18.208815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:18.208953 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:18.209466 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:36:18.209471 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:36:18.209627 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 08:36:18.209774 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 08:36:18.210052 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 08:36:18.210438 6370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:18.197585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54014 (local address 127.15.134.253:36201)
0928 08:36:18.197765 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:18.197774 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:18.197863 (+ 89us) server_negotiation.cc:407] Connection header received
0928 08:36:18.198019 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:18.198022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:18.198083 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:18.198202 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:18.198908 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.199570 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.200794 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.201217 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.201354 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:18.204350 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:18.204360 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:18.204366 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:18.204418 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:18.207868 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:18.208486 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:18.208491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:18.208495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:18.208688 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:18.209064 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:18.209070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:18.209074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:18.209342 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:18.209502 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:18.209879 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 08:36:18.210078 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 08:36:18.211431 6440 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:36:18 dist-test-slave-qp2n krb5kdc[6211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584578, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.252@KRBTEST.COM for kudu/127.15.134.252@KRBTEST.COM
Sep 28 08:36:18 dist-test-slave-qp2n krb5kdc[6211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.15.134.253@KRBTEST.COM
I20260928 08:36:18.226281 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:18.212142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:44707 (local address 127.0.0.1:55602)
0928 08:36:18.212539 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:18.212558 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:36:18.212695 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:36:18.213059 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:36:18.213062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:36:18.213420 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:36:18.213544 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:18.213551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.214813 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.214820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:18.216090 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:18.216103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.217288 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.217295 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:18.217503 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:18.218147 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:36:18.218195 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:36:18.220100 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:36:18.223471 (+ 3371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:18.223474 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:18.223476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:18.223997 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:18.224358 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:18.224364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:18.224368 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:18.224550 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:18.225135 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:36:18.225139 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:36:18.225329 (+ 190us) client_negotiation.cc:769] Sending connection context
0928 08:36:18.225469 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 08:36:18.225795 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":1}
I20260928 08:36:18.226280 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:18.212174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55602 (local address 127.15.134.252:44707)
0928 08:36:18.212359 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:18.212365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:18.212646 (+ 281us) server_negotiation.cc:407] Connection header received
0928 08:36:18.212830 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:18.212833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:18.212915 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:18.213009 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:18.213689 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.214671 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.216241 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.217043 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.217179 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:18.220285 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:18.220291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:18.220299 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:18.220338 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:18.223293 (+ 2955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:18.224112 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:18.224120 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:18.224124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:18.224254 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:18.224726 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:18.224731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:18.224734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:18.225014 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:18.225161 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:18.225554 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 08:36:18.225947 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":164,"server-negotiator.queue_time_us":60}
I20260928 08:36:18.227733 6370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:18.212373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54026 (local address 127.15.134.253:36201)
0928 08:36:18.212513 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:18.212518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:18.212540 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:36:18.212749 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:18.212756 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:18.212835 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:18.212901 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:18.214430 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.215602 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.217214 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.217876 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.218015 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:18.222223 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:18.222228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:18.222231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:18.222261 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:18.225111 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:18.225820 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:18.225825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:18.225830 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:18.225969 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:18.226386 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:18.226393 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:18.226396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:18.226743 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:18.227034 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:18.227284 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 08:36:18.227458 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 08:36:18.232296 6440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04ba0230e2a94fa4ac1565f5fd59538c: Bootstrap starting.
I20260928 08:36:18.232728 15899 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:36:18.235764 6440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04ba0230e2a94fa4ac1565f5fd59538c: Neither blocks nor log segments found. Creating new log.
I20260928 08:36:18.236929 6440 log.cc:824] T 00000000000000000000000000000000 P 04ba0230e2a94fa4ac1565f5fd59538c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:36:18.235097 6266 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d9a2100921884f22b34d30cf770baf46"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50390
I20260928 08:36:18.240131 6440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04ba0230e2a94fa4ac1565f5fd59538c: No bootstrap required, opened a new log
I20260928 08:36:18.248327 6440 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04ba0230e2a94fa4ac1565f5fd59538c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2100921884f22b34d30cf770baf46" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38047 } } peers { permanent_uuid: "36ce736df30d45989958f89c7229f69e" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 } } peers { permanent_uuid: "04ba0230e2a94fa4ac1565f5fd59538c" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } }
I20260928 08:36:18.248741 6440 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04ba0230e2a94fa4ac1565f5fd59538c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04ba0230e2a94fa4ac1565f5fd59538c, State: Initialized, Role: FOLLOWER
I20260928 08:36:18.249203 6440 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04ba0230e2a94fa4ac1565f5fd59538c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2100921884f22b34d30cf770baf46" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38047 } } peers { permanent_uuid: "36ce736df30d45989958f89c7229f69e" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 } } peers { permanent_uuid: "04ba0230e2a94fa4ac1565f5fd59538c" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } }
I20260928 08:36:18.250392 6448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ba0230e2a94fa4ac1565f5fd59538c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2100921884f22b34d30cf770baf46" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38047 } } peers { permanent_uuid: "36ce736df30d45989958f89c7229f69e" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 } } peers { permanent_uuid: "04ba0230e2a94fa4ac1565f5fd59538c" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } } }
I20260928 08:36:18.250984 6448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ba0230e2a94fa4ac1565f5fd59538c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:36:18.251750 6440 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04ba0230e2a94fa4ac1565f5fd59538c [sys.catalog]: configured and running, proceeding with master startup.
W20260928 08:36:18.262930 6459 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04ba0230e2a94fa4ac1565f5fd59538c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:36:18.263127 6459 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:36:18.263384 6291 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } attempt: 4
I20260928 08:36:18.260505 6373 master_runner.cc:186] Error getting master registration for 127.15.134.254:38047: OK, instance_id { permanent_uuid: "d9a2100921884f22b34d30cf770baf46" instance_seqno: 1790584577123686 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 08:36:18.271276 6373 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 28 08:36:18 dist-test-slave-qp2n krb5kdc[6211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584577, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.254@KRBTEST.COM for kudu/127.15.134.252@KRBTEST.COM
I20260928 08:36:18.281226 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:18.264225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55618 (local address 127.15.134.252:44707)
0928 08:36:18.264433 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:18.264440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:18.264461 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:36:18.264527 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:18.264532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:18.264647 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:18.264754 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:18.265880 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.267175 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.269047 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.269885 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.270017 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:18.274202 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:18.274224 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:18.274231 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:18.274275 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:18.278084 (+ 3809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:18.278917 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:18.278921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:18.278925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:18.279084 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:18.279510 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:18.279515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:18.279518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:18.279976 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:18.280193 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:18.280603 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 08:36:18.280848 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 08:36:18.281234 6293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:18.264012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:44707 (local address 127.0.0.1:55618)
0928 08:36:18.264232 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:18.264254 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:36:18.264375 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:36:18.264939 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:36:18.264944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:36:18.265514 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:36:18.265725 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:18.265737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.267339 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.267344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:18.268890 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:18.268907 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.270057 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.270063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:18.270287 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:18.271004 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:36:18.271031 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:36:18.273964 (+ 2933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:36:18.278249 (+ 4285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:18.278256 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:18.278261 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:18.278796 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:18.279217 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:18.279222 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:18.279226 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:18.279398 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:18.280115 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:36:18.280120 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:36:18.280329 (+ 209us) client_negotiation.cc:769] Sending connection context
0928 08:36:18.280496 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 08:36:18.280818 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 08:36:18.286444 6291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46: Bootstrap starting.
I20260928 08:36:18.290768 6291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46: Neither blocks nor log segments found. Creating new log.
I20260928 08:36:18.292068 6291 log.cc:824] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46: Log is configured to *not* fsync() on all Append() calls
I20260928 08:36:18.295625 6291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46: No bootstrap required, opened a new log
I20260928 08:36:18.303454 6291 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2100921884f22b34d30cf770baf46" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38047 } } peers { permanent_uuid: "36ce736df30d45989958f89c7229f69e" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 } } peers { permanent_uuid: "04ba0230e2a94fa4ac1565f5fd59538c" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } }
I20260928 08:36:18.303917 6291 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9a2100921884f22b34d30cf770baf46, State: Initialized, Role: FOLLOWER
I20260928 08:36:18.304450 6291 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2100921884f22b34d30cf770baf46" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38047 } } peers { permanent_uuid: "36ce736df30d45989958f89c7229f69e" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 } } peers { permanent_uuid: "04ba0230e2a94fa4ac1565f5fd59538c" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } }
I20260928 08:36:18.305827 6462 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2100921884f22b34d30cf770baf46" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38047 } } peers { permanent_uuid: "36ce736df30d45989958f89c7229f69e" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 } } peers { permanent_uuid: "04ba0230e2a94fa4ac1565f5fd59538c" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } } }
I20260928 08:36:18.306527 6462 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:36:18.307317 6291 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:36:18.307422 6266 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 08:36:18.307628 6266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:36:18.312114 6266 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2100921884f22b34d30cf770baf46" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38047 } } peers { permanent_uuid: "36ce736df30d45989958f89c7229f69e" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 } } peers { permanent_uuid: "04ba0230e2a94fa4ac1565f5fd59538c" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } }
I20260928 08:36:18.315219 6266 leader_election.cc:288] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46 [CANDIDATE]: Term 1 election: Requested vote from peers 36ce736df30d45989958f89c7229f69e (127.15.134.253:36201), 04ba0230e2a94fa4ac1565f5fd59538c (127.15.134.252:44707)
I20260928 08:36:18.315511 6415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9a2100921884f22b34d30cf770baf46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "04ba0230e2a94fa4ac1565f5fd59538c"
I20260928 08:36:18.316148 6415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04ba0230e2a94fa4ac1565f5fd59538c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:36:18.320725 6415 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04ba0230e2a94fa4ac1565f5fd59538c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9a2100921884f22b34d30cf770baf46 in term 1.
I20260928 08:36:18.321605 6243 leader_election.cc:302] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04ba0230e2a94fa4ac1565f5fd59538c, d9a2100921884f22b34d30cf770baf46; no voters:
I20260928 08:36:18.322351 6462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:36:18.329541 6462 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46 [term 1 LEADER]: Becoming Leader. State: Replica: d9a2100921884f22b34d30cf770baf46, State: Running, Role: LEADER
I20260928 08:36:18.330345 6462 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2100921884f22b34d30cf770baf46" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38047 } } peers { permanent_uuid: "36ce736df30d45989958f89c7229f69e" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 } } peers { permanent_uuid: "04ba0230e2a94fa4ac1565f5fd59538c" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } }
I20260928 08:36:18.338063 6472 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9a2100921884f22b34d30cf770baf46. Latest consensus state: current_term: 1 leader_uuid: "d9a2100921884f22b34d30cf770baf46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2100921884f22b34d30cf770baf46" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38047 } } peers { permanent_uuid: "36ce736df30d45989958f89c7229f69e" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 } } peers { permanent_uuid: "04ba0230e2a94fa4ac1565f5fd59538c" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } } }
I20260928 08:36:18.338527 6472 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46 [sys.catalog]: This master's current role is: LEADER
I20260928 08:36:18.343824 6478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:36:18.360939 6478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:36:18.361630 6366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } attempt: 5
W20260928 08:36:18.362967 6245 tablet.cc:2369] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:36:18.390586 6394 tablet.cc:2369] T 00000000000000000000000000000000 P 04ba0230e2a94fa4ac1565f5fd59538c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:36:18.408311 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:18.342841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50394 (local address 127.15.134.254:38047)
0928 08:36:18.343147 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:18.343156 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:18.349214 (+ 6058us) server_negotiation.cc:407] Connection header received
0928 08:36:18.351778 (+ 2564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:18.351786 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:18.351879 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:18.351962 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:18.360858 (+ 8896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.361993 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.373216 (+ 11223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.374092 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.374211 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:18.387390 (+ 13179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:18.387400 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:18.387405 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:18.387453 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:18.395268 (+ 7815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:18.400540 (+ 5272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:18.400548 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:18.400551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:18.400774 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:18.402838 (+ 2064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:18.402844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:18.402848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:18.403204 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:18.403466 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:18.407260 (+ 3794us) server_negotiation.cc:299] Negotiation successful
0928 08:36:18.407568 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:36:18.410295 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:18.349787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55632 (local address 127.15.134.252:44707)
0928 08:36:18.349976 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:18.349982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:18.358147 (+ 8165us) server_negotiation.cc:407] Connection header received
0928 08:36:18.358411 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:18.358416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:18.358517 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:18.358603 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:18.364339 (+ 5736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.365812 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.379100 (+ 13288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.379892 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.380017 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:18.396959 (+ 16942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:18.396966 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:18.396972 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:18.397016 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:18.400970 (+ 3954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:18.403966 (+ 2996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:18.403971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:18.403974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:18.404141 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:18.404807 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:18.404813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:18.404816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:18.405204 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:18.405495 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:18.409660 (+ 4165us) server_negotiation.cc:299] Negotiation successful
0928 08:36:18.409892 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 08:36:18 dist-test-slave-qp2n krb5kdc[6211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790584577, etypes {rep=17 tkt=17 ses=17}, kudu/127.15.134.253@KRBTEST.COM for kudu/127.15.134.252@KRBTEST.COM
I20260928 08:36:18.411376 6370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:18.345069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54036 (local address 127.15.134.253:36201)
0928 08:36:18.345267 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:18.345273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:18.352869 (+ 7596us) server_negotiation.cc:407] Connection header received
0928 08:36:18.355203 (+ 2334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:18.355210 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:18.355305 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:18.355383 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:18.368161 (+ 12778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.369553 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.376750 (+ 7197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.377495 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.377625 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:18.389138 (+ 11513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:18.389144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:18.389148 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:18.389192 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:18.394440 (+ 5248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:18.401968 (+ 7528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:18.401973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:18.401976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:18.402199 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:18.405965 (+ 3766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:18.405973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:18.405979 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:18.406414 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:18.406696 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:18.410718 (+ 4022us) server_negotiation.cc:299] Negotiation successful
0928 08:36:18.410973 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 08:36:18.425148 6415 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04ba0230e2a94fa4ac1565f5fd59538c [term 1 FOLLOWER]: Refusing update from remote peer d9a2100921884f22b34d30cf770baf46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:36:18.426816 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:18.362251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.134.252:44707 (local address 127.0.0.1:55646)
0928 08:36:18.362658 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0928 08:36:18.362680 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:36:18.362787 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:36:18.389102 (+ 26315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:36:18.389110 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:36:18.389645 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:36:18.389910 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:18.389926 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.401927 (+ 12001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.401937 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:18.403477 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:36:18.403496 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.407403 (+ 3907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.407410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:36:18.407651 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:18.408340 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:36:18.408365 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:36:18.412036 (+ 3671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:36:18.422038 (+ 10002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:18.422046 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:18.422052 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:18.422954 (+ 902us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:18.423690 (+ 736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:36:18.423696 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:36:18.423700 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:36:18.424044 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:36:18.425098 (+ 1054us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:36:18.425103 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:36:18.425514 (+ 411us) client_negotiation.cc:769] Sending connection context
0928 08:36:18.425728 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 08:36:18.426235 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276}
I20260928 08:36:18.427891 6462 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04ba0230e2a94fa4ac1565f5fd59538c" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:36:18.428896 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:18.363019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55646 (local address 127.15.134.252:44707)
0928 08:36:18.387039 (+ 24020us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:18.387213 (+ 174us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:18.387404 (+ 191us) server_negotiation.cc:407] Connection header received
0928 08:36:18.387638 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:18.387644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:18.388537 (+ 893us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:18.391023 (+ 2486us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:18.393068 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.401539 (+ 8471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.403962 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:18.407109 (+ 3147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:18.412262 (+ 5153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:18.413946 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:36:18.413955 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:36:18.414127 (+ 172us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:36:18.414270 (+ 143us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:36:18.421768 (+ 7498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:18.423223 (+ 1455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:18.423230 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:18.423234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:18.423489 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:36:18.424273 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:36:18.424301 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:36:18.424306 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:36:18.424867 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:36:18.427536 (+ 2669us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:18.427747 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 08:36:18.428090 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23189,"thread_start_us":22042,"threads_started":1}
I20260928 08:36:18.436015 6366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36ce736df30d45989958f89c7229f69e: Bootstrap starting.
I20260928 08:36:18.443078 6366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36ce736df30d45989958f89c7229f69e: Neither blocks nor log segments found. Creating new log.
I20260928 08:36:18.445552 6366 log.cc:824] T 00000000000000000000000000000000 P 36ce736df30d45989958f89c7229f69e: Log is configured to *not* fsync() on all Append() calls
I20260928 08:36:18.456388 6366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36ce736df30d45989958f89c7229f69e: No bootstrap required, opened a new log
I20260928 08:36:18.429605 6224 master_runner.cc:186] Error getting master registration for 127.15.134.253:36201: OK, instance_id { permanent_uuid: "36ce736df30d45989958f89c7229f69e" instance_seqno: 1790584577638404 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 08:36:18.459048 6224 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 08:36:18.470010 6366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36ce736df30d45989958f89c7229f69e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2100921884f22b34d30cf770baf46" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38047 } } peers { permanent_uuid: "36ce736df30d45989958f89c7229f69e" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 } } peers { permanent_uuid: "04ba0230e2a94fa4ac1565f5fd59538c" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } }
I20260928 08:36:18.470556 6366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36ce736df30d45989958f89c7229f69e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ce736df30d45989958f89c7229f69e, State: Initialized, Role: FOLLOWER
I20260928 08:36:18.470606 6448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ba0230e2a94fa4ac1565f5fd59538c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9a2100921884f22b34d30cf770baf46. Latest consensus state: current_term: 1 leader_uuid: "d9a2100921884f22b34d30cf770baf46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2100921884f22b34d30cf770baf46" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38047 } } peers { permanent_uuid: "36ce736df30d45989958f89c7229f69e" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 } } peers { permanent_uuid: "04ba0230e2a94fa4ac1565f5fd59538c" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } } }
I20260928 08:36:18.471102 6448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ba0230e2a94fa4ac1565f5fd59538c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:36:18.471278 6366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36ce736df30d45989958f89c7229f69e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2100921884f22b34d30cf770baf46" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38047 } } peers { permanent_uuid: "36ce736df30d45989958f89c7229f69e" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 } } peers { permanent_uuid: "04ba0230e2a94fa4ac1565f5fd59538c" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } }
I20260928 08:36:18.473487 6490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36ce736df30d45989958f89c7229f69e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2100921884f22b34d30cf770baf46" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38047 } } peers { permanent_uuid: "36ce736df30d45989958f89c7229f69e" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 } } peers { permanent_uuid: "04ba0230e2a94fa4ac1565f5fd59538c" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } } }
I20260928 08:36:18.474386 6490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36ce736df30d45989958f89c7229f69e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:36:18.476315 6486 mvcc.cc:205] Tried to move back new op lower bound from 7334234433195102208 to 7334234432848654336. Current Snapshot: MvccSnapshot[applied={T|T < 7334234433195102208}]
I20260928 08:36:18.480664 6487 mvcc.cc:205] Tried to move back new op lower bound from 7334234433195102208 to 7334234432848654336. Current Snapshot: MvccSnapshot[applied={T|T < 7334234433195102208}]
I20260928 08:36:18.482105 6462 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d9a2100921884f22b34d30cf770baf46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2100921884f22b34d30cf770baf46" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38047 } } peers { permanent_uuid: "36ce736df30d45989958f89c7229f69e" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 } } peers { permanent_uuid: "04ba0230e2a94fa4ac1565f5fd59538c" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } } }
I20260928 08:36:18.482491 6462 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46 [sys.catalog]: This master's current role is: LEADER
I20260928 08:36:18.483682 6478 catalog_manager.cc:1347] Generated new cluster ID: 8dbcdefffff54b65b3a7d4f8e0a8f9dc
I20260928 08:36:18.483788 6478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:36:18.486021 6366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36ce736df30d45989958f89c7229f69e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:36:18.488183 6341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d9a2100921884f22b34d30cf770baf46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "36ce736df30d45989958f89c7229f69e"
I20260928 08:36:18.488696 6341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36ce736df30d45989958f89c7229f69e [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:36:18.492312 6341 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 36ce736df30d45989958f89c7229f69e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9a2100921884f22b34d30cf770baf46 in term 1.
I20260928 08:36:18.493487 6340 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 36ce736df30d45989958f89c7229f69e [term 1 FOLLOWER]: Refusing update from remote peer d9a2100921884f22b34d30cf770baf46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:36:18.487716 6448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ba0230e2a94fa4ac1565f5fd59538c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d9a2100921884f22b34d30cf770baf46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2100921884f22b34d30cf770baf46" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38047 } } peers { permanent_uuid: "36ce736df30d45989958f89c7229f69e" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 } } peers { permanent_uuid: "04ba0230e2a94fa4ac1565f5fd59538c" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } } }
I20260928 08:36:18.494287 6448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ba0230e2a94fa4ac1565f5fd59538c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:36:18.501583 6472 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36ce736df30d45989958f89c7229f69e" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:36:18.540793 6478 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:36:18.543336 6478 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:36:18.552913 6490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36ce736df30d45989958f89c7229f69e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9a2100921884f22b34d30cf770baf46. Latest consensus state: current_term: 1 leader_uuid: "d9a2100921884f22b34d30cf770baf46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2100921884f22b34d30cf770baf46" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38047 } } peers { permanent_uuid: "36ce736df30d45989958f89c7229f69e" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 } } peers { permanent_uuid: "04ba0230e2a94fa4ac1565f5fd59538c" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } } }
I20260928 08:36:18.553359 6490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36ce736df30d45989958f89c7229f69e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:36:18.553802 6499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36ce736df30d45989958f89c7229f69e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d9a2100921884f22b34d30cf770baf46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2100921884f22b34d30cf770baf46" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38047 } } peers { permanent_uuid: "36ce736df30d45989958f89c7229f69e" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 } } peers { permanent_uuid: "04ba0230e2a94fa4ac1565f5fd59538c" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } } }
I20260928 08:36:18.554241 6499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36ce736df30d45989958f89c7229f69e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:36:18.556044 6462 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d9a2100921884f22b34d30cf770baf46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9a2100921884f22b34d30cf770baf46" member_type: VOTER last_known_addr { host: "127.15.134.254" port: 38047 } } peers { permanent_uuid: "36ce736df30d45989958f89c7229f69e" member_type: VOTER last_known_addr { host: "127.15.134.253" port: 36201 } } peers { permanent_uuid: "04ba0230e2a94fa4ac1565f5fd59538c" member_type: VOTER last_known_addr { host: "127.15.134.252" port: 44707 } } }
I20260928 08:36:18.556339 6462 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46 [sys.catalog]: This master's current role is: LEADER
I20260928 08:36:18.577512 6478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9a2100921884f22b34d30cf770baf46: Generated new TSK 0
I20260928 08:36:18.578980 6478 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:36:18.599102 6507 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260928 08:36:18.688035 6446 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:36:18.677405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50396 (local address 127.15.134.254:38047)
0928 08:36:18.677617 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:18.677625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:18.685528 (+ 7903us) server_negotiation.cc:407] Connection header received
0928 08:36:18.686066 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:18.686073 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:18.686198 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:18.686312 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:18.687705 (+ 1393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50396: BlockingRecv error: recv got EOF from 127.0.0.1:50396 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 08:36:18.691246 6370 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:36:18.676946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54042 (local address 127.15.134.253:36201)
0928 08:36:18.677180 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:18.677187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:18.689456 (+ 12269us) server_negotiation.cc:407] Connection header received
0928 08:36:18.689625 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:18.689631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:18.689756 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:18.689861 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:18.690998 (+ 1137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54042: BlockingRecv error: recv got EOF from 127.0.0.1:54042 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260928 08:36:18.692732 6484 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:36:18.685025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55650 (local address 127.15.134.252:44707)
0928 08:36:18.685190 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:18.685195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:18.690320 (+ 5125us) server_negotiation.cc:407] Connection header received
0928 08:36:18.690836 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:18.690843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:18.690957 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:18.691028 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:18.692454 (+ 1426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55650: BlockingRecv error: recv got EOF from 127.0.0.1:55650 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 08:36:18.699394 6370 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:36:18.697130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54058 (local address 127.15.134.253:36201)
0928 08:36:18.697317 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:18.697325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:18.697859 (+ 534us) server_negotiation.cc:407] Connection header received
0928 08:36:18.698061 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:18.698067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:18.698213 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:18.698325 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:18.699136 (+ 811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54058: BlockingRecv error: recv got EOF from 127.0.0.1:54058 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 08:36:18.711980 6370 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:36:18.703726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54064 (local address 127.15.134.253:36201)
0928 08:36:18.703910 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:18.703916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:18.710464 (+ 6548us) server_negotiation.cc:407] Connection header received
0928 08:36:18.710684 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:18.710690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:18.710786 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:18.710900 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:18.711753 (+ 853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54064: BlockingRecv error: recv got EOF from 127.0.0.1:54064 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 08:36:18.722777 6370 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:36:18.717949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54076 (local address 127.15.134.253:36201)
0928 08:36:18.718129 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:18.718136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:18.721213 (+ 3077us) server_negotiation.cc:407] Connection header received
0928 08:36:18.721409 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:18.721415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:18.721528 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:18.721650 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:18.722529 (+ 879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54076: BlockingRecv error: recv got EOF from 127.0.0.1:54076 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 08:36:18.737515 6370 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:36:18.735074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54080 (local address 127.15.134.253:36201)
0928 08:36:18.735332 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:18.735339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:18.735852 (+ 513us) server_negotiation.cc:407] Connection header received
0928 08:36:18.736077 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:18.736084 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:18.736184 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:18.736298 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:18.737278 (+ 980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54080: BlockingRecv error: recv got EOF from 127.0.0.1:54080 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
W20260928 08:36:18.756963 6370 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:36:18.754266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54084 (local address 127.15.134.253:36201)
0928 08:36:18.754495 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:18.754502 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:18.755128 (+ 626us) server_negotiation.cc:407] Connection header received
0928 08:36:18.755459 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:18.755465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:18.755577 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:18.755724 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:18.756706 (+ 982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54084: BlockingRecv error: recv got EOF from 127.0.0.1:54084 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260928 08:36:18.779281 6370 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:36:18.776814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54090 (local address 127.15.134.253:36201)
0928 08:36:18.777030 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:18.777037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:18.777655 (+ 618us) server_negotiation.cc:407] Connection header received
0928 08:36:18.777863 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:18.777868 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:18.777961 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:18.778070 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:18.779048 (+ 978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54090: BlockingRecv error: recv got EOF from 127.0.0.1:54090 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 08:36:18.820748 6370 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:36:18.818440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54104 (local address 127.15.134.253:36201)
0928 08:36:18.818625 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:18.818632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:18.819163 (+ 531us) server_negotiation.cc:407] Connection header received
0928 08:36:18.819368 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:18.819376 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:18.819478 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:18.819596 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:18.820434 (+ 838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54104: BlockingRecv error: recv got EOF from 127.0.0.1:54104 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 08:36:18.890931 6370 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:36:18.888080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54114 (local address 127.15.134.253:36201)
0928 08:36:18.888320 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:18.888327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:18.889097 (+ 770us) server_negotiation.cc:407] Connection header received
0928 08:36:18.889358 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:18.889362 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:18.889443 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:18.889537 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:18.890699 (+ 1162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54114: BlockingRecv error: recv got EOF from 127.0.0.1:54114 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260928 08:36:19.025395 6370 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:36:19.023136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54118 (local address 127.15.134.253:36201)
0928 08:36:19.023385 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:19.023394 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:19.023790 (+ 396us) server_negotiation.cc:407] Connection header received
0928 08:36:19.024021 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:19.024029 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:19.024134 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:19.024254 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:19.025080 (+ 826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54118: BlockingRecv error: recv got EOF from 127.0.0.1:54118 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 08:36:19.265439 6459 catalog_manager.cc:1259] Loaded cluster ID: 8dbcdefffff54b65b3a7d4f8e0a8f9dc
I20260928 08:36:19.265560 6459 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 04ba0230e2a94fa4ac1565f5fd59538c: loading cluster ID for follower catalog manager: success
I20260928 08:36:19.268236 6459 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 04ba0230e2a94fa4ac1565f5fd59538c: acquiring CA information for follower catalog manager: success
I20260928 08:36:19.270350 6459 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 04ba0230e2a94fa4ac1565f5fd59538c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 08:36:19.289867 6370 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:36:19.287847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54122 (local address 127.15.134.253:36201)
0928 08:36:19.288120 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:19.288126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:19.288319 (+ 193us) server_negotiation.cc:407] Connection header received
0928 08:36:19.288473 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:19.288477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:19.288713 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:19.288802 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:36:19.289694 (+ 892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54122: BlockingRecv error: recv got EOF from 127.0.0.1:54122 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 08:36:19.603595 6507 catalog_manager.cc:1259] Loaded cluster ID: 8dbcdefffff54b65b3a7d4f8e0a8f9dc
I20260928 08:36:19.603724 6507 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 36ce736df30d45989958f89c7229f69e: loading cluster ID for follower catalog manager: success
I20260928 08:36:19.606685 6507 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 36ce736df30d45989958f89c7229f69e: acquiring CA information for follower catalog manager: success
I20260928 08:36:19.608897 6507 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 36ce736df30d45989958f89c7229f69e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:36:19.816931 6575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:19.809391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54138 (local address 127.15.134.253:36201)
0928 08:36:19.810205 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:19.810212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:19.810236 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:36:19.810311 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:19.810317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:19.810493 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:19.810628 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:36:19.811237 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:19.812370 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:19.813911 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:19.814684 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:19.814824 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:19.815314 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:36:19.816064 (+ 750us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:36:19.816251 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:19.816398 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 08:36:19.816499 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":534,"threads_started":1}
I20260928 08:36:19.833426 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:19.824516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55654 (local address 127.15.134.252:44707)
0928 08:36:19.825135 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:19.825142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:19.825600 (+ 458us) server_negotiation.cc:407] Connection header received
0928 08:36:19.825819 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:19.825826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:19.826020 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:19.826206 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:36:19.826851 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:19.827993 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:19.829660 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:19.830301 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:19.830438 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:19.831394 (+ 956us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:36:19.832190 (+ 796us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:36:19.832349 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:19.832603 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 08:36:19.832766 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":365,"threads_started":1}
I20260928 08:36:19.861600 6588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:19.840978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50398 (local address 127.15.134.254:38047)
0928 08:36:19.853515 (+ 12537us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:19.853520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:19.853537 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:36:19.853639 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:19.853643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:19.853830 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:19.854147 (+ 317us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:36:19.854925 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:19.856207 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:19.858433 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:19.859148 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:19.859274 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:19.860002 (+ 728us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:36:19.860489 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:36:19.860593 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:19.861157 (+ 564us) server_negotiation.cc:299] Negotiation successful
0928 08:36:19.861251 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12414,"thread_start_us":12250,"threads_started":1}
I20260928 08:36:19.861914 6575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:19.840513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54142 (local address 127.15.134.253:36201)
0928 08:36:19.840773 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:19.840782 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:19.854926 (+ 14144us) server_negotiation.cc:407] Connection header received
0928 08:36:19.855217 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:19.855223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:19.855315 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:19.855385 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:36:19.856349 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:19.857519 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:19.859694 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:19.860308 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:19.860403 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:19.861102 (+ 699us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:36:19.861246 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:36:19.861308 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:19.861518 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 08:36:19.861610 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 08:36:19.867365 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 6224
I20260928 08:36:19.868723 6581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:36:19.855237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55668 (local address 127.15.134.252:44707)
0928 08:36:19.855654 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0928 08:36:19.855660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:36:19.862154 (+ 6494us) server_negotiation.cc:407] Connection header received
0928 08:36:19.862434 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:36:19.862440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:36:19.862537 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:36:19.862603 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:36:19.863171 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:19.864233 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:19.866098 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:36:19.866715 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:36:19.866810 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:36:19.867740 (+ 930us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:36:19.867881 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:36:19.867934 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 08:36:19.868362 (+ 428us) server_negotiation.cc:299] Negotiation successful
0928 08:36:19.868444 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260928 08:36:19.869187 6286 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:36:19.988871 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 6224
W20260928 08:36:20.010948 6317 connection.cc:535] server connection from 127.0.0.1:54008 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 08:36:20.011579 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 6298
I20260928 08:36:20.012835 6361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:36:20.121268 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 6298
I20260928 08:36:20.142935 15899 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task0uZKZe/build/asan/bin/kudu pid 6373
I20260928 08:36:20.144173 6435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:36:20.254125 15899 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task0uZKZe/build/asan/bin/kudu with pid 6373
2026-09-28T08:36:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5039 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9981 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (374937 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST